diff --git a/README.md b/README.md index 62152f34d..f1a7c6a0e 100644 --- a/README.md +++ b/README.md @@ -181,11 +181,11 @@ To undo it, remove the entries in Windows Security → App & browser control → | **Video**⁵: MP4, MOV, MKV, WEBM, AVI, WMV, FLV, MPEG, Video CD (DAT), TS, M2TS, 3GP, OGV, VOB, …;
Wisenet WAVE / Nx Witness exports (NOV) read | MP4 (H.264, HEVC, AV1), WEBM, MOV, MKV, AVI, GIF, APNG, MP3, M4A, 720p, split into parts;
subtitle tracks kept (MKV, MP4, MOV, WebM);
subtitles (SRT, ASS, WebVTT) | | **Audio**⁵: MP3, M4A, AAC, WAV, FLAC, OGG, OPUS, WMA, AIFF, AMR, APE, WavPack, TTA, DSD (DSF, DFF), DTS, AC3, E-AC3, …;
tracker modules (XM, IT, S3M) read | MP3, M4A (AAC, ALAC), M4R (iPhone ringtone), WAV, FLAC, OGG, OPUS, AAC, WavPack, TTA, W64, AC3, E-AC3 | | **Subtitles**: SRT, ASS/SSA, WebVTT, SAMI (SMI), LRC, KRC, TTML, MicroDVD, SCC | each other as UTF-8 (Korean CP949 files read correctly) | -| **Archives**: ZIP, ANSYS Workbench (WBPZ), 7Z, RAR, TAR, TAR.GZ, TAR.BZ2, TAR.XZ, GZ, ISO, CAB, DMG, ALZ, EGG, …;
JAR, NuGet (NUPKG), Android packages (APKS, XAPK, APKM, AAB), Adobe extensions (ZXP) read | extract, ZIP, 7Z, TAR, TAR.GZ;
any files → one ZIP; folders → ZIP, ISO (Joliet or UDF 2.01) | +| **Archives**: ZIP, ANSYS Workbench (WBPZ), 7Z, RAR, TAR, TAR.GZ, TAR.BZ2, TAR.XZ, GZ, ISO, CAB, DMG, ALZ, EGG, Garry's Mod addon (GMA), Valve pak (VPK), files packed into a Source map (BSP), …;
JAR, NuGet (NUPKG), Android packages (APKS, XAPK, APKM, AAB), Adobe extensions (ZXP) read | extract, ZIP, 7Z, TAR, TAR.GZ, GMA;
any files → one ZIP; folders → ZIP, ISO (Joliet or UDF 2.01), GMA | | **CAD**: DWG, DXF
Gerber PCB layers (GBR, GTL, GBL, GTO, …) read;
G-code of 3D printers and CNC machines (GCODE, GCO, NGC) read | PDF, SVG, PNG, DWG ↔ DXF;
G-code → toolpath (from above or isometric) or the slicer's preview | | **Fonts**: TTF, OTF, WOFF, WOFF2, EOT;
FON (Windows bitmap fonts) read | each other;
FON → one font per size | | **Data**: JSON, JSON Lines, YAML, TOML, XML
vCard, iCalendar | each other, CSV and XLSX (tables);
vCard, iCalendar ↔ CSV | -| **3D models**: GLB, glTF, STL, OBJ (+ MTL), 3MF, PLY;
FBX read | each other, PNG preview | +| **3D models**: GLB, glTF, STL, OBJ (+ MTL), 3MF, PLY, studiomdl data (SMD);
FBX, Source engine model (MDL, with its VTF textures) read | each other, PNG preview | | **Geospatial vectors**: Shapefile, GeoJSON, KML, KMZ, GPX
CSV with coordinates; Garmin FIT, OpenStreetMap (.osm) read | each other and CSV / XLSX / JSON;
projected data (UTM, Web Mercator, Lambert, Albers) comes out as WGS 84 longitude / latitude | Unmarked formats work out of the box, without Office, Hancom Office or LibreOffice. diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 5b5d0b5d4..5702122de 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -33,7 +33,7 @@ Filee chooses engines automatically. Settings → *Engines* shows their status, | Pictures to 3D relief | built in | **PNG, JPG, WEBP, … (every picture ImageMagick reads) → GLB, glTF, STL, OBJ, 3MF, PLY**: a relief, brighter areas higher | | **Geospatial vector data** | built in | **GeoJSON, KML, KMZ, GPX, Shapefile ↔ each other and CSV** (longitude / latitude or WKT); Garmin FIT (activities) and OpenStreetMap XML (`.osm`), both read only, → each of them and CSV; projected Shapefiles (UTM, Web Mercator, Lambert, Albers) are converted to WGS 84; through CSV to XLSX, JSON, … (see below) | | rhwp | bundled with the installer (`engines/rhwp`, with the Microsoft C++ runtime, see below) | HWP/HWPX → PDF, **HWP → HWPX, HWPX → HWP** | -| Archives (7-Zip) | bundled with the installer (`engines/7zip`, ~2.5 MB) + built-in readers | ZIP (including ANSYS Workbench WBPZ), 7Z, RAR, TAR (+ GZ/BZ2/XZ/Z/7Z/LZ), CAB, ISO, DMG, split (.001), Minecraft and Modrinth packs, … **→ folder, ZIP, 7Z, TAR, TAR.GZ/BZ2/XZ, the packs**; ALZ, EGG, lzip read in-process; "Compress into one archive" for any files | +| Archives (7-Zip) | bundled with the installer (`engines/7zip`, ~2.5 MB) + built-in readers | ZIP (including ANSYS Workbench WBPZ), 7Z, RAR, TAR (+ GZ/BZ2/XZ/Z/7Z/LZ), CAB, ISO, DMG, split (.001), Minecraft and Modrinth packs, Garry's Mod addons (GMA), Valve paks (VPK), the files packed into Source maps (BSP), … **→ folder, ZIP, 7Z, TAR, TAR.GZ/BZ2/XZ, the packs, GMA**; ALZ, EGG, lzip, GMA, VPK read in-process; "Compress into one archive" for any files | | LibreOffice + H2Orestart + Java | **optional download** (~420 MB, 1.3 GB on disk) | older and rare formats only: DOC, XLS, PPT, RTF and OpenDocument → PDF and to each other, output as DOCX/ODT/ODS/ODP; HWP/HWPX → DOCX/ODT; read-only formats (see below) → PDF and their kind's editable formats; FODT, OpenDocument templates (OTT, OTS, OTP) and XLT out; SVG → EMF/WMF/ODG, a PDF page → SVG | | Pandoc | **optional download** (~42 MB, 240 MB on disk) | Markdown ↔ DOCX/ODT/RTF, DOCX/ODT/HTML/RTF → Markdown, HTML ↔ DOCX/ODT, markup languages (reStructuredText, LaTeX, Org, AsciiDoc, Typst, MediaWiki, DocBook, Jupyter, Textile, man, OPML) ↔ Markdown/HTML/DOCX/ODT and each other (and → RTF, EPUB, TXT), Markdown and markup → PPTX slides | | Ghostscript | **optional download** (~20 MB, 31 MB on disk) | EPS/PS → PDF, PDF → EPS/PS, PostScript-based AI → PDF (images through PDF and PDFium); smaller PDFs when a preset asks for it; PDF/A-2b for PDF/A presets; renders CMYK PDF/AI pages for CMYK JPG/TIFF | @@ -1421,8 +1421,8 @@ to draw). G-code is a read-only CAD format (fields: 3D models and CAD). ## 3D models -`Filee.Engines/Models3D` converts GLB, glTF, STL, OBJ, 3MF and PLY into each other, and FBX into any of them, in -managed code: every reader +`Filee.Engines/Models3D` converts GLB, glTF, STL, OBJ, 3MF and PLY into each other, FBX and Source engine models (MDL) +into any of them, and studiomdl data (SMD) to and from them, in managed code: every reader fills one in-memory model (`Model3D`: parts with positions, normals, UVs, vertex colours and triangles; materials with a base colour and a base colour texture) and every writer takes it, so each pair is one step. glTF and GLB are read and written with [SharpGLTF](https://github.com/vpenades/SharpGLTF) (MIT); STL, OBJ/MTL, PLY, 3MF and the FBX @@ -1456,6 +1456,8 @@ What each format keeps: | OBJ + MTL | `o` / `g` / `usemtl` as parts, `v` (+ r g b), `vt`, `vn`, faces of any size; `mtllib` with `Kd`, `d` / `Tr`, `map_Kd` (options skipped) next to the file | **the `.mtl` and the textures are written next to the `.obj`** (`name.obj`, `name.mtl`, `name_texture.png`, named like the output); vertex colours as `v x y z r g b` | | 3MF | objects, components (also in other model parts, as Bambu Studio writes them), build transforms, base materials, colour groups (materials extension); `support` / `other` objects skipped | one object per part, base materials from material colours, a colour group from vertex colours | | FBX (read only) | binary 6.x–7.x (64-bit records from 7.5, zlib-compressed arrays) and ASCII; mesh geometry with polygons of any size, the first normal, UV, colour and material layer (by polygon vertex, control point or polygon, or all the same; direct or indexed); the model hierarchy from the connections with translation, rotation (every Euler order, pre- / post-rotation, pivots and offsets), scaling and geometric transforms; property templates from Definitions; material `DiffuseColor` and `Opacity`; the diffuse texture embedded (Video `Content`) or named next to the file; animation, skins, blend shapes, NURBS, cameras and lights skipped (the rest pose is read) | — (not written) | +| SMD (studiomdl data) | the triangles of a reference SMD, one part per material (bones and weights not kept) | from an MDL: its bones, bind pose and weights (see below); from the others: a static prop with one root bone | +| MDL (Source engine, read only) | versions 44–49 with the `.vvd` and `.dx90.vtx` (else `.vtx`, `.dx80.vtx`, `.sw.vtx`) beside it (see below) | — (needs Valve's studiomdl) | | PLY | ASCII, binary little / big endian; vertex position, normal, UV (`s`/`t`, `u`/`v`, `texture_u`/`texture_v`), colours of any type; face lists with per-corner `texcoord` and face colours; `comment TextureFile`; other elements skipped | binary little endian, one mesh; material colours become vertex colours | Lost on the way (no place in the target): **STL** keeps only the triangles (no colours, UVs, normals or materials); @@ -1470,6 +1472,35 @@ FBX's texture names) are read only from the model's folder or a folder below it. points anywhere else is looked up by its name in the model's folder instead, so a downloaded model cannot pull a picture from elsewhere on the disk into its output. +**Source engine models** (`.mdl`, #1526, the Games field; `Models3D/SourceModel.cs`): Garry's Mod, Counter-Strike: +Source, Team Fortress 2 and Half-Life 2 models are three files studiomdl writes side by side: the `.mdl` (bones, +textures, texture folders, the skin table, body parts → models → meshes), the `.vvd` (vertices with up to three bone +weights; fixups say which runs make LOD 0) and the `.dx90.vtx` (per mesh, strip groups of triangle lists or strips +indexing the mesh's vertices). All three must have the `.mdl`'s checksum, else the files belong to different builds and +the model is refused; a missing `.vvd` or `.vtx` is named. The model is read in its bind pose with its default body +groups (the first model of each body part), the first skin and LOD 0; animations, flexes, attachments and the physics +mesh (`.phy`) are not read. Source is Z up and measured in inches (25.4 mm), its triangles are clockwise and its UVs +start at the top left, so the model is turned to Filee's Y up, millimetres, counter-clockwise and bottom-left UVs; each +mesh becomes a part named after its model, with one material per texture. Later branches' 8-byte-longer strip groups +(CS:GO) are read when the indices of the usual layout don't fit. +The textures are found as Garry's Mod finds them: in the addon's or game's own layout, the `materials` folder beside the +`models` folder the model is in, `materials//.vmt` for each of the model's texture folders +(`$cdmaterials`), letter case aside, then the material's `$basetexture` (also through a `patch` material's `include`) as +a `.vtf`, decoded by the VTF reader into a PNG texture; `$nocull` makes the material double-sided. Only `.vmt` and +`.vtf` files below that `materials` folder are read, never through `..`. A model outside such a layout, or one whose +materials come from the game itself (the HL2 textures), keeps plain materials. OBJ output writes the textures beside the +`.obj`, GLB embeds them. + +**Studiomdl data** (`.smd`, #1526, the Games field; `Models3D/SmdFormat.cs`): the text Valve's studiomdl compiles models +from (`nodes`, a `skeleton` at `time 0`, `triangles`: a material line and three vertex lines of parent bone, position, +normal, UV and bone weights), Z up, in inches, counter-clockwise, UVs from the bottom left. An MDL becomes a reference +SMD as Crowbar decompiles it: its bones with their bind pose (position and rotation relative to the parent), its +triangles with up to three bone weights per vertex and the textures as materials, read straight from the model so the +skeleton is kept (Filee's own 3D model has no place for it). Any other model becomes a static prop: one `root` bone, +every vertex weighted to it, the material's (or texture's) name as the material line, normals computed where the model +has none. Reading takes the triangles of a reference SMD as a static model; an animation SMD (no triangles) is refused +with "no triangles", a broken line with its number. The "SMD" preset is in the palette, after glTF. + **3D → pictures**: a small software rasteriser (`MeshRenderer`) draws an isometric view from the front right, above, with one light and flat shading per triangle, vertex colours or the texture interpolated across it, and a z-buffer, at 2048 px and scaled down to 1024 px for smooth edges, on a transparent background. The preset's image options @@ -1824,6 +1855,9 @@ bundled with the installer), never a 7-Zip installed on the PC. It runs as a sep | Split archives (.001, .7z.001, .zip.001, .tar.001) | 7-Zip (combining .002, .003 … automatically) | — | | ALZ (ALZip) | Filee: store, deflate and ALZip's bzip2 variant | — | | EGG (ALZip) | Filee: store, deflate, bzip2, LZMA (decoded by 7-Zip); several blocks per file | — | +| GMA (Garry's Mod addon) | Filee: versions 1–3, CRCs checked, an `addon.json` written beside the files | Filee, as gmad does (see below) | +| BSP (Source engine map), its packed files | Filee copies the pakfile lump (a ZIP) out, 7-Zip unpacks it | — | +| VPK (Valve pak) | Filee: versions 1 and 2, `_dir.vpk` with its `_000.vpk` … parts, CRCs checked; a PS Vita `.vpk` (a ZIP) by 7-Zip | — | - **Extract** (target `folder`): the archive is unpacked into a hidden folder next to the result and moved into place when complete. When the archive holds exactly one top folder, its content becomes the result folder (`photos.zip` @@ -1837,6 +1871,38 @@ bundled with the installer), never a 7-Zip installed on the PC. It runs as a sep an archive or a folder whose only top folder holds the marker gets that folder's content, not the folder (`GamePacks.Root`); an add-on (MCADDON, several packs side by side) is packed as it is. The presets (MCPACK … MRPACK) are in the palette only. +- **Garry's Mod addons** (`.gma`, #1526, the Games field; `Archives/Gma.cs`, Filee's own reader): the Steam + Workshop's format, a header (title, a JSON description with the type and tags, author, timestamp), a file list + (names, sizes, CRC-32s) and the files one after another, ended by a CRC-32 of everything before it. Every name is + checked before anything is written (an absolute name, a drive or a `..` stops the addon), the entries are capped at + the archive limit and their sizes must fit inside the addon. Unpacking checks every file's CRC (0, as `gmad -nocrc` + writes it, is not checked) and writes an `addon.json` with the title, description, type and tags, so the folder packs + back into the same addon; the required-content list and the author are not kept (Garry's Mod uses neither). Packing + a folder, a ZIP of one or the files dropped onto "Compress into one archive" reads `addon.json` like gmad: `title` + (else the folder's name), `type` (else `servercontent`), `tags`, `description` and the `ignore` wildcards. Only + files on gmad's whitelist go in (`lua/*.lua`, `materials/*.vtf`, `models/*.mdl` …, the ones Garry's Mod mounts; + `addon.json`, `thumbs.db`, `desktop.ini`, `.git*` and `.DS_Store` never), so a `.psd` or a `readme.txt` beside them + is left out rather than failing the addon as gmad does without `-warninvalid`; a folder with none of them is + refused. Names are stored in lower case, sorted, with a CRC each. The type and tags are written as given (lower + case), not checked against the Workshop's lists. The "GMA" preset is in the palette after the game packs (schema + 96), not on a donut. +- **Valve paks** (`.vpk`, #1526, the Games field; `Archives/Vpk.cs`): the archives Source engine games keep their files + in (`hl2_textures_dir.vpk`, the content Garry's Mod mounts). The directory file holds a tree of extensions, folders + and names; each file's content is a few preload bytes stored in the tree plus its data, which follows the tree in the + directory file (archive index 0x7FFF) or sits in `_nnn.vpk` next to it. The directory file is what unpacks; + a data part on its own is refused with the name of its directory file, a missing part names the part. Every name is + checked before anything is written, the tree is capped at 256 MB and the entries at the archive limit, preload bytes + must lie inside the tree, parts are `_000` to `_999`, and each file's data must lie inside its part; the total is + checked against the free space first. The MD5 and signature sections of version 2 are not read; every file's CRC-32 + is checked. Nothing writes VPK (Garry's Mod + loads addons as folders or GMA). A PS Vita package has the same extension but is a ZIP: it goes to 7-Zip. +- **Source engine maps** (`.bsp`, #1526, the Games field; `Archives/Bsp.cs`): the custom textures, models and sounds + packed into a map (by VBSP, VIDE or bspzip) are a ZIP stored as lump 40, the pakfile. Filee finds it in the lump + directory (offset, length, version, code; Left 4 Dead 2's version, offset, length order is used when only that one + points inside the file), copies it to the work folder and 7-Zip unpacks it, so a map extracts or converts to any + archive like one, with the same checks as any ZIP (names leaving the folder, the entry limit, the free space). The + lump must lie inside the map, and its copy is bounded by the lump's length. A map without packed files (an empty ZIP), a compressed pakfile lump and a Quake or GoldSrc map + (no `VBSP`) are refused with a message saying so. The map's geometry is not read. - **Adobe extension packages** (`.zxp`, #1382, read only like JAR): the panels and plug-ins of After Effects, Premiere Pro, Photoshop and the other Creative Cloud apps are signed ZIPs (`mimetype`, the extension's files and `META-INF/signatures.xml`). They extract and convert to every archive target with the signature file inside. diff --git a/src/Filee.Core/Formats/FormatFields.cs b/src/Filee.Core/Formats/FormatFields.cs index 500ed61c3..a09650158 100644 --- a/src/Filee.Core/Formats/FormatFields.cs +++ b/src/Filee.Core/Formats/FormatFields.cs @@ -83,7 +83,7 @@ .. Enum.GetValues().Select(category => category.ToString().ToLow "dts", "tar", "tgz", "tbz2", "txz", "tar7z", "tlz", "gz", "bz2", "xz", "lz", "lzma", "jar", "zxp", "nupkg", "apks", "aab", "mcpack", "mcaddon", "mcworld", "mctemplate", "mrpack", "cab", "gerber", "cpio", "deb", "rpm", "dwg", "ttf", "otf", "woff", "woff2", "ass", "ssa", "ttml", "scc", "glb", "gltf", - "stl", "obj", "3mf", "ply", "fbx", "gcode", "geojson", "kml", "kmz", "gpx", "shp", "fit", "osm", "xm", "it", "ani", "vtf" + "stl", "obj", "3mf", "ply", "fbx", "gcode", "geojson", "kml", "kmz", "gpx", "shp", "fit", "osm", "xm", "it", "ani", "vtf", "gma", "vpk", "bsp", "mdl", "smd" ]), (FormatGrade.Legacy, [ @@ -100,7 +100,7 @@ .. Enum.GetValues().Select(category => category.ToString().ToLow // Layout, print, photo post-production, signs and embroidery. (["psd", "psb", "exr", "hdr", "xcf", "kra", "pdn", "raw", "eps", "ps", "ai", "cdr", "plt", "dst"], ["design"]), // Game textures and cutscenes, Minecraft packs and modpacks. - (["dds", "vtf", "usm", "bik", "mcpack", "mcaddon", "mcworld", "mctemplate", "mrpack"], ["game"]), + (["dds", "vtf", "usm", "bik", "mcpack", "mcaddon", "mcworld", "mctemplate", "mrpack", "gma", "vpk", "bsp", "mdl", "smd"], ["game"]), // Medical imaging and astronomy. (["dcm", "fits"], ["science"]), // Software work: raw image formats of graphics tools, app icons and cursors, config and data files, flat XML diff --git a/src/Filee.Core/Formats/FormatRegistry.cs b/src/Filee.Core/Formats/FormatRegistry.cs index 86df633b1..f3bb049a6 100644 --- a/src/Filee.Core/Formats/FormatRegistry.cs +++ b/src/Filee.Core/Formats/FormatRegistry.cs @@ -355,6 +355,14 @@ public static class FormatRegistry new("mcworld", "Minecraft world", FormatCategory.Archive, ["mcworld"]), new("mctemplate", "Minecraft world template", FormatCategory.Archive, ["mctemplate"]), new("mrpack", "Modrinth modpack", FormatCategory.Archive, ["mrpack"]), + // Garry's Mod addons (Steam Workshop), read and written by Filee as gmad does: addon.json gives the title, type, + // tags and ignores, and only the files Garry's Mod loads go in. + new("gma", "Garry's Mod addon (GMA)", FormatCategory.Archive, ["gma"]), + // Valve paks (Source engine games' files, "pak01_dir.vpk" with its "pak01_000.vpk" … parts), read by Filee; a PS + // Vita package under the same extension is a ZIP and goes to 7-Zip. + new("vpk", "Valve pak (VPK)", FormatCategory.Archive, ["vpk"], Writable: false), + // Source engine maps: the files packed into a map (its pakfile lump, a ZIP) unpack like an archive. + new("bsp", "Source map (BSP)", FormatCategory.Archive, ["bsp"], Writable: false), new("cab", "CAB", FormatCategory.Archive, ["cab"], Writable: false), new("cpio", "CPIO", FormatCategory.Archive, ["cpio"], Writable: false), new("deb", "DEB", FormatCategory.Archive, ["deb"], Writable: false), @@ -409,6 +417,11 @@ public static class FormatRegistry new("3mf", "3MF", FormatCategory.Model3D, ["3mf"]), new("ply", "PLY", FormatCategory.Model3D, ["ply"]), new("fbx", "FBX", FormatCategory.Model3D, ["fbx"], Writable: false), + // Source engine models (Garry's Mod and other Source games), read with the .vvd and .vtx beside them and their + // textures from the "materials" folder above them. + new("mdl", "Source model (MDL)", FormatCategory.Model3D, ["mdl"], Writable: false), + // Studiomdl data, the text the Source engine's model compiler reads (from an MDL with its bones and weights). + new("smd", "Studiomdl data (SMD)", FormatCategory.Model3D, ["smd"]), // Geospatial vector data (CSV with coordinates is a Data format and reaches these through the geo engine) new("geojson", "GeoJSON", FormatCategory.Geo, ["geojson"]), diff --git a/src/Filee.Core/Presets/BuiltInData.cs b/src/Filee.Core/Presets/BuiltInData.cs index 7fe755d92..c3b591686 100644 --- a/src/Filee.Core/Presets/BuiltInData.cs +++ b/src/Filee.Core/Presets/BuiltInData.cs @@ -160,6 +160,7 @@ .. CatalogPresets(), .. DataPresets(), .. CodecAndSubtitlePresets(), .. ModelPresets(), + .. SourceModelPresets(), .. GeoPresets(), .. TracePresets(), ]; @@ -226,8 +227,8 @@ public static List ClipPresets() => ]; /// - /// Minecraft packs and Modrinth modpacks (#824, #898, added to the library in settings schema 62): in the palette - /// only, after "TAR", in the Games field (off in a new install). + /// Minecraft packs and Modrinth modpacks (#824, #898, added to the library in settings schema 62) and Garry's Mod + /// addons (#1526, schema 96): in the palette only, after "TAR", in the Games field (off in a new install). /// public static List GamePackPresets() => [ @@ -236,6 +237,7 @@ public static List GamePackPresets() => new() { Id = "to-mcworld", Name = "MCWORLD", TargetFormat = "mcworld" }, new() { Id = "to-mctemplate", Name = "MCTEMPLATE", TargetFormat = "mctemplate" }, new() { Id = "to-mrpack", Name = "MRPACK", TargetFormat = "mrpack" }, + new() { Id = "to-gma", Name = "GMA", TargetFormat = "gma" }, ]; /// @@ -590,6 +592,15 @@ public static List ModelPresets() => new() { Id = "to-gltf", Name = "glTF", TargetFormat = "gltf" }, ]; + /// + /// Studiomdl data for Source engine models (#1526, added to the library in settings schema 97): in the palette only, + /// after glTF, in the Games field (off in a new install). + /// + public static List SourceModelPresets() => + [ + new() { Id = "to-smd", Name = "SMD", TargetFormat = "smd" }, + ]; + /// /// Geospatial vector targets (added in settings schema 25, see SettingsMigrations). CSV, the sixth slice of the /// donut, is the spreadsheet preset. KMZ is a KML in a ZIP and is on no donut. diff --git a/src/Filee.Core/Settings/AppSettings.cs b/src/Filee.Core/Settings/AppSettings.cs index da022ae84..aa112de3d 100644 --- a/src/Filee.Core/Settings/AppSettings.cs +++ b/src/Filee.Core/Settings/AppSettings.cs @@ -170,7 +170,7 @@ public static bool IsFlipKeyAllowed(string key, IEnumerable gest public sealed class AppSettings { /// Current schema version. Increase when the format changes and add a migration. - public const int CurrentSchemaVersion = 94; + public const int CurrentSchemaVersion = 97; public int SchemaVersion { get; set; } = CurrentSchemaVersion; diff --git a/src/Filee.Core/Settings/UserDataStore.cs b/src/Filee.Core/Settings/UserDataStore.cs index f72e1b754..d2b689c07 100644 --- a/src/Filee.Core/Settings/UserDataStore.cs +++ b/src/Filee.Core/Settings/UserDataStore.cs @@ -1176,6 +1176,23 @@ public static bool ApplyToLibrary(int from, List presets, List presets) return changed; } + /// + /// "GMA" after the last game pack of the library (else after "TAR", else at the end); the game packs the user removed + /// stay removed. + /// + private static bool AddGma(List presets) + { + if (presets.Any(p => p.Id == "to-gma")) + return false; + var packs = BuiltInData.GamePackPresets().Select(p => p.Id).ToHashSet(); + var at = presets.FindLastIndex(p => packs.Contains(p.Id)); + if (at < 0) + at = presets.FindIndex(p => p.Id == "to-tar"); + presets.Insert(at < 0 ? presets.Count : at + 1, BuiltInData.GamePackPresets().Single(p => p.Id == "to-gma")); + return true; + } + + /// "SMD" after the last 3D model preset of the library (glTF), or at the end. + private static bool AddSourcePresets(List presets) + { + var changed = false; + foreach (var preset in BuiltInData.SourceModelPresets().Where(p => presets.All(existing => existing.Id != p.Id))) + { + var models = BuiltInData.ModelPresets().Select(p => p.Id).ToHashSet(); + var last = presets.FindLastIndex(p => models.Contains(p.Id)); + presets.Insert(last < 0 ? presets.Count : last + 1, preset); + changed = true; + } + return changed; + } + private static bool AddCropVideo(List presets) { var at = presets.FindIndex(p => p.Id == "tool-clip"); diff --git a/src/Filee.Engines/Archives/ArchiveConverter.cs b/src/Filee.Engines/Archives/ArchiveConverter.cs index 1ec9258aa..c7bf17ed4 100644 --- a/src/Filee.Engines/Archives/ArchiveConverter.cs +++ b/src/Filee.Engines/Archives/ArchiveConverter.cs @@ -1,6 +1,6 @@ // Archives: unpack (to a folder, the "Extract" preset), convert between archive formats (ZIP, 7Z, TAR, TAR.GZ, -// TAR.BZ2, TAR.XZ, single-file GZ / BZ2 / XZ, and the Minecraft and Modrinth packs that are ZIPs, see GamePacks), and -// "Compress into one archive" for any files (IFileCombiner). +// TAR.BZ2, TAR.XZ, single-file GZ / BZ2 / XZ, the Minecraft and Modrinth packs that are ZIPs, see GamePacks, and Garry's +// Mod addons, see Gma), and "Compress into one archive" for any files (IFileCombiner). // Reading uses the bundled 7-Zip (engines/7zip, LGPL, separate process) plus Filee's own ALZ, EGG and lzip readers; // ZIP, TAR and TAR.GZ are written in-process, the other formats by 7-Zip. See docs/ENGINES.md. @@ -16,7 +16,7 @@ namespace Filee.Engines.Archives; public sealed class ArchiveConverter : IConverter, IFileCombiner { /// Archive formats Filee writes (plus ). - private static readonly string[] ArchiveTargets = ["zip", "7z", "tar", "tgz", "tbz2", "txz", .. GamePacks.Formats]; + private static readonly string[] ArchiveTargets = ["zip", "7z", "tar", "tgz", "tbz2", "txz", "gma", .. GamePacks.Formats]; /// Compressed single files (no names or folders inside); they convert among each other. private static readonly string[] SingleFileSources = ["gz", "bz2", "xz", "lzma", "z", "lz"]; @@ -38,7 +38,7 @@ .. from source in SingleFileSources from target in ArchivePacker.SingleFileFormats select new ConversionEdge(source, target), .. ArchiveExtractor.ReadableFormats.Select(source => new ConversionEdge(source, "iso", DirectOnly: true)), - .. new[] { "zip", "7z", "tar", "tgz", "tbz2", "txz" } + .. new[] { "zip", "7z", "tar", "tgz", "tbz2", "txz", "gma" } .Select(target => new ConversionEdge(FormatRegistry.Folder, target, DirectOnly: true)), ]; @@ -85,6 +85,8 @@ public async Task> ConvertAsync(ConversionStep step, IProg } if (GamePacks.Formats.Contains(step.To)) unpacked = GamePacks.Root(unpacked, step.To); + else if (step.To == "gma") + unpacked = Gma.Root(unpacked); var items = Directory.EnumerateFileSystemEntries(unpacked) .Order(StringComparer.Ordinal) .Select(path => new PackItem(path, Path.GetFileName(path))) @@ -141,10 +143,11 @@ public async Task CombineAsync(IReadOnlyList inputPaths, string outputPa string workDirectory, IProgress? progress, CancellationToken cancellationToken) { var sevenZip = ArchivePacker.BuiltInFormats.Contains(format) ? null : _sevenZip ??= SevenZip.Locate(); - // A pack made from its folder holds the folder's content, so its marker file is at the top. - var entries = GamePacks.Formats.Contains(format) && inputPaths is [var folder] && Directory.Exists(folder) - ? Directory.EnumerateFileSystemEntries(GamePacks.Root(folder, format)).Order(StringComparer.Ordinal) - .Select(path => new PackItem(path, Path.GetFileName(path))).ToList() + // A pack (or a Garry's Mod addon) made from its folder holds the folder's content, so its marker file (addon.json) + // is at the top. + var entries = (GamePacks.Formats.Contains(format) || format == "gma") && inputPaths is [var folder] && Directory.Exists(folder) + ? Directory.EnumerateFileSystemEntries(format == "gma" ? Gma.Root(folder) : GamePacks.Root(folder, format)) + .Order(StringComparer.Ordinal).Select(path => new PackItem(path, Path.GetFileName(path))).ToList() : UniqueEntries(inputPaths); await new ArchivePacker(sevenZip).PackAsync(entries, outputPath, format, preset.Archive.Level, workDirectory, progress, cancellationToken); diff --git a/src/Filee.Engines/Archives/ArchiveExtractor.cs b/src/Filee.Engines/Archives/ArchiveExtractor.cs index 66a57e109..405245f65 100644 --- a/src/Filee.Engines/Archives/ArchiveExtractor.cs +++ b/src/Filee.Engines/Archives/ArchiveExtractor.cs @@ -1,6 +1,8 @@ // Unpacks every archive format Filee reads into a folder: the bundled 7-Zip for most of them, Filee's own readers for -// ALZ, EGG and lzip. Compressed TARs (.tar.gz, .tar.lz …) and RPM payloads are unpacked stage by stage until the -// files appear. Every stage is checked first (encryption, paths leaving the folder, disk space) and links are removed. +// ALZ, EGG, lzip, Garry's Mod addons (GMA) and Valve paks (VPK); the files packed into a Source engine map (BSP) are a +// ZIP that Filee copies out for 7-Zip. Compressed TARs (.tar.gz, .tar.lz …) and RPM payloads are unpacked stage by +// stage until the files appear. Every stage is checked first (encryption, paths leaving the folder, disk space) and +// links are removed. // A single compressed stream (.gz, .bz2 …) is written by Filee as 7-Zip decodes it, so it can't fill the drive. The // archive's Mark of the Web goes onto the unpacked files. @@ -25,7 +27,7 @@ internal sealed class ArchiveExtractor(SevenZip? sevenZip, Func? s ]; /// Formats read by Filee's own code (lzip still needs 7-Zip's LZMA decoder). - public static readonly IReadOnlyList BuiltInFormats = ["alz", "egg", "lz", "tlz"]; + public static readonly IReadOnlyList BuiltInFormats = ["alz", "egg", "lz", "tlz", "gma", "vpk", "bsp"]; /// Everything that can be unpacked. public static IEnumerable ReadableFormats => SevenZipFormats.Concat(BuiltInFormats); @@ -85,6 +87,34 @@ private async Task ExtractOnceAsync(string archive, string format, string folder await EggReader.ExtractAsync(archive, folder, sevenZip, scratchRoot, cancellationToken); PassOnMark(archive, folder); break; + case "gma": + await Task.Run(() => Gma.Extract(archive, folder, cancellationToken), cancellationToken); + PassOnMark(archive, folder); + break; + // A PS Vita package is a ZIP under the same extension. + case "vpk" when Vpk.IsZip(archive): + await ExtractOnceAsync(archive, "zip", folder, scratchRoot, cancellationToken); + break; + case "vpk": + await Task.Run(() => Vpk.Extract(archive, folder, cancellationToken), cancellationToken); + PassOnMark(archive, folder); + break; + case "bsp": + { + // The map's packed files are a ZIP inside it: copied out, then unpacked like any ZIP. + var pakfile = Path.Combine(scratchRoot, $"pakfile-{Guid.NewGuid():N}.zip"); + try + { + await Task.Run(() => Bsp.ExtractPakfile(archive, pakfile, cancellationToken), cancellationToken); + await ExtractOnceAsync(pakfile, "zip", folder, scratchRoot, cancellationToken); + } + finally + { + ArchiveFiles.TryDeleteFile(pakfile); + } + PassOnMark(archive, folder); + break; + } case "lz": case "tlz": var output = FormatRegistry.NameWithoutExtension(archive) + (format == "tlz" ? ".tar" : ""); diff --git a/src/Filee.Engines/Archives/ArchivePacker.cs b/src/Filee.Engines/Archives/ArchivePacker.cs index 442d1fe26..8172a99ae 100644 --- a/src/Filee.Engines/Archives/ArchivePacker.cs +++ b/src/Filee.Engines/Archives/ArchivePacker.cs @@ -1,10 +1,12 @@ -// Writes archives. ZIP (and the game packs that are ZIPs, see GamePacks), TAR and TAR.GZ are written in-process (System.IO.Compression / System.Formats.Tar), so file +// Writes archives. ZIP (and the game packs that are ZIPs, see GamePacks), TAR, TAR.GZ and Garry's Mod addons (GMA, see Gma) +// are written in-process (System.IO.Compression / System.Formats.Tar / Filee's own writer), so file // names are exactly what Filee chooses: UTF-8 in ZIP (with the language encoding flag), PAX headers in TAR. 7Z, the // bzip2 / xz TAR variants and single-file GZ / BZ2 / XZ are written by the bundled 7-Zip. using System.Formats.Tar; using System.IO.Compression; using System.Text; +using Filee.Core.Formats; using Filee.Core.Presets; using Filee.Engines.Infrastructure; @@ -18,7 +20,7 @@ internal sealed class ArchivePacker(SevenZip? sevenZip) { /// Formats written without 7-Zip. public static readonly IReadOnlySet BuiltInFormats = - new HashSet(["zip", "tar", "tgz", .. GamePacks.Formats], StringComparer.OrdinalIgnoreCase); + new HashSet(["zip", "tar", "tgz", "gma", .. GamePacks.Formats], StringComparer.OrdinalIgnoreCase); /// Formats that hold exactly one file, without name or folders. public static readonly IReadOnlySet SingleFileFormats = new HashSet(StringComparer.OrdinalIgnoreCase) { "gz", "bz2", "xz" }; @@ -41,6 +43,12 @@ public async Task PackAsync(IReadOnlyList items, string outputPath, st case "zip" or "mcpack" or "mcaddon" or "mcworld" or "mctemplate" or "mrpack": await Task.Run(() => WriteZip(items, temp, level, progress, cancellationToken), cancellationToken); break; + case "gma": + // The addon's title, without an addon.json, is the name it is saved under (its folder's name). + await Task.Run(() => Gma.Write( + [.. Expand(items).Where(f => !f.IsFolder).Select(f => (f.EntryName, f.Path))], temp, + FormatRegistry.NameWithoutExtension(outputPath), progress, cancellationToken), cancellationToken); + break; case "tar": case "tgz": await Task.Run(() => WriteTar(items, temp, format == "tgz" ? level : null, progress, cancellationToken), cancellationToken); diff --git a/src/Filee.Engines/Archives/Bsp.cs b/src/Filee.Engines/Archives/Bsp.cs new file mode 100644 index 000000000..141aa8750 --- /dev/null +++ b/src/Filee.Engines/Archives/Bsp.cs @@ -0,0 +1,65 @@ +// Source engine maps (.bsp): the files packed into a map (custom textures, models and sounds; the "pakfile" lump, made by +// VBSP and tools such as VIDE or bspzip) are a ZIP stored inside the map. Filee copies that ZIP out and 7-Zip unpacks it. +// +// "VBSP" + version i32 (19/20 in Garry's Mod, up to 21), then 64 lumps of 16 bytes and the map revision; +// a lump is offset i32, length i32, version i32 and a four-byte code (its uncompressed size when it is compressed). +// Left 4 Dead 2 writes version, offset, length instead; the order that points inside the file is used. +// Lump 40 is the pakfile. + +using System.Buffers.Binary; + +namespace Filee.Engines.Archives; + +/// Takes the packed files out of Source engine maps. +internal static class Bsp +{ + private const int LumpCount = 64; + private const int PakfileLump = 40; + + /// "VBSP", the version and the lump directory (the map revision follows). + private const int HeaderSize = 8 + LumpCount * 16 + 4; + + /// An empty ZIP: just its end record. + private const int EmptyZip = 22; + + /// Where the pakfile lump of is. + internal static (long Offset, long Length) Pakfile(Stream map, string mapName) + { + Span header = stackalloc byte[HeaderSize]; + if (map.ReadAtLeast(header, HeaderSize, throwOnEndOfStream: false) < HeaderSize || !header.StartsWith("VBSP"u8)) + throw new InvalidDataException($"{mapName} is not a Source engine map (VBSP)."); + var version = BinaryPrimitives.ReadInt32LittleEndian(header[4..]); + var lump = header.Slice(8 + PakfileLump * 16, 16); + long a = BinaryPrimitives.ReadInt32LittleEndian(lump); + long b = BinaryPrimitives.ReadInt32LittleEndian(lump[4..]); + long c = BinaryPrimitives.ReadInt32LittleEndian(lump[8..]); + var compressed = BinaryPrimitives.ReadInt32LittleEndian(lump[12..]) != 0; + + bool Inside(long offset, long length) => length == 0 || offset >= HeaderSize && length > 0 && offset + length <= map.Length; + (long Offset, long Length) found; + if (Inside(a, b)) + found = (a, b); + else if (version >= 21 && Inside(b, c)) // Left 4 Dead 2: version, offset, length + found = (b, c); + else + throw ArchiveSafety.Damaged(mapName, "the packed files point outside the map"); + + if (found.Length <= EmptyZip) + throw new InvalidDataException($"{mapName} has no files packed into the map; its textures and models come from the game or an addon."); + if (compressed) + throw new InvalidDataException($"{mapName} keeps its packed files compressed (LZMA lump), which Filee does not unpack."); + return found; + } + + /// Copies the packed files' ZIP of to . + public static void ExtractPakfile(string map, string zipPath, CancellationToken cancellationToken) + { + var mapName = Path.GetFileName(map); + using var stream = new FileStream(map, FileMode.Open, FileAccess.Read, FileShare.Read, 1 << 16); + var (offset, length) = Pakfile(stream, mapName); + stream.Position = offset; + using var output = new FileStream(zipPath, FileMode.CreateNew, FileAccess.Write, FileShare.None, 1 << 16); + if (ArchiveFiles.CopyLimited(new BoundedStream(stream, length), output, length, cancellationToken) != length) + throw ArchiveSafety.Damaged(mapName, "the packed files are cut off"); + } +} diff --git a/src/Filee.Engines/Archives/Gma.cs b/src/Filee.Engines/Archives/Gma.cs new file mode 100644 index 000000000..bbc9e839d --- /dev/null +++ b/src/Filee.Engines/Archives/Gma.cs @@ -0,0 +1,450 @@ +// Garry's Mod addons (.gma), the format of the Steam Workshop: read and written by Filee, as Facepunch's gmad does. +// Layout (little endian, from gmad's AddonFormat.h, AddonReader.h and create_gmad.cpp): +// +// "GMAD" + version (1 byte, 1–3; gmad writes 3) +// SteamID u64 (unused, 0) + timestamp u64 (Unix seconds) +// version > 1: required content, NUL-terminated strings ending with an empty one +// title, description, author (NUL-terminated); addon version i32 (unused, 1) +// file list: per file a number u32 (1, 2, …), its name (NUL-terminated, lower case), size i64 and CRC-32 u32; +// a number 0 ends the list +// the files' content one after another, then a CRC-32 of everything before it (0 when gmad ran with -nocrc) +// +// The description is the JSON gmad builds from addon.json ({"description", "type", "tags"}). Unpacking writes an +// addon.json beside the files, so the folder packs back into the same addon. Packing reads it like gmad: title, type, +// tags and the "ignore" wildcards; files outside gmad's whitelist (the only ones Garry's Mod loads) are left out, and +// names are stored in lower case. + +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; + +namespace Filee.Engines.Archives; + +/// Unpacks and writes Garry's Mod addons. +internal static class Gma +{ + private static ReadOnlySpan Ident => "GMAD"u8; + + /// The version gmad writes. + private const byte Version = 3; + + /// The settings file of an addon folder; never packed, written back on unpacking. + public const string AddonJson = "addon.json"; + + /// + /// gmad's whitelist (AddonWhiteList.h), in order: the files Garry's Mod mounts from an addon. A "!" entry takes back + /// what the entries before it allowed. "*" matches any characters, slashes included. + /// + private static readonly string[] WhiteList = + [ + "lua/*.lua", "scenes/*.vcd", "particles/*.pcf", "resource/fonts/*.ttf", "scripts/vehicles/*.txt", + "resource/localization/*/*.properties", "maps/*.bsp", "maps/*.lmp", "maps/*.nav", "maps/*.ain", "maps/thumb/*.png", + "sound/*.wav", "sound/*.mp3", "sound/*.ogg", "materials/*.vmt", "materials/*.vtf", "materials/*.png", + "materials/*.jpg", "materials/*.jpeg", "materials/colorcorrection/*.raw", "models/*.mdl", "models/*.phy", + "models/*.ani", "models/*.vvd", "models/*.vtx", "!models/*.sw.vtx", "!models/*.360.vtx", "!models/*.xbox.vtx", + "gamemodes/*/*.txt", "!gamemodes/*/*/*.txt", "gamemodes/*/*.fgd", "!gamemodes/*/*/*.fgd", "gamemodes/*/logo.png", + "gamemodes/*/icon24.png", "gamemodes/*/gamemode/*.lua", "gamemodes/*/entities/effects/*.lua", + "gamemodes/*/entities/weapons/*.lua", "gamemodes/*/entities/entities/*.lua", "gamemodes/*/backgrounds/*.png", + "gamemodes/*/backgrounds/*.jpg", "gamemodes/*/backgrounds/*.jpeg", "gamemodes/*/content/models/*.mdl", + "gamemodes/*/content/models/*.phy", "gamemodes/*/content/models/*.ani", "gamemodes/*/content/models/*.vvd", + "gamemodes/*/content/models/*.vtx", "!gamemodes/*/content/models/*.sw.vtx", "!gamemodes/*/content/models/*.360.vtx", + "!gamemodes/*/content/models/*.xbox.vtx", "gamemodes/*/content/materials/*.vmt", "gamemodes/*/content/materials/*.vtf", + "gamemodes/*/content/materials/*.png", "gamemodes/*/content/materials/*.jpg", "gamemodes/*/content/materials/*.jpeg", + "gamemodes/*/content/materials/colorcorrection/*.raw", "gamemodes/*/content/scenes/*.vcd", + "gamemodes/*/content/particles/*.pcf", "gamemodes/*/content/resource/fonts/*.ttf", + "gamemodes/*/content/scripts/vehicles/*.txt", "gamemodes/*/content/resource/localization/*/*.properties", + "gamemodes/*/content/maps/*.bsp", "gamemodes/*/content/maps/*.nav", "gamemodes/*/content/maps/*.ain", + "gamemodes/*/content/maps/thumb/*.png", "gamemodes/*/content/sound/*.wav", "gamemodes/*/content/sound/*.mp3", + "gamemodes/*/content/sound/*.ogg", "data_static/*.txt", "data_static/*.dat", "data_static/*.json", + "data_static/*.xml", "data_static/*.csv", "shaders/fxc/*.vcs", + ]; + + /// The longest name Filee reads from a file list (gmad reserves 180 bytes; longer is a damaged file). + private const int MaxNameBytes = 4096; + + /// One file of an addon. + internal sealed record Entry(string Name, long Size, uint Crc, long Offset); + + /// What the header says about an addon. + internal sealed record Header(string Title, string Description, string? Type, IReadOnlyList Tags, string Author, + DateTimeOffset? Created, IReadOnlyList Entries); + + /// True for a file that starts like an addon. + public static bool IsGma(ReadOnlySpan start) => start.StartsWith(Ident); + + /// + /// Unpacks every file of into , with an addon.json for + /// its title, description, type and tags. + /// + public static void Extract(string archive, string targetDirectory, CancellationToken cancellationToken) + { + var archiveName = Path.GetFileName(archive); + using var stream = new FileStream(archive, FileMode.Open, FileAccess.Read, FileShare.Read, 1 << 16); + var header = ReadHeader(stream, archiveName); + // Every name is checked before anything is written: one that leaves the folder stops the whole addon. + foreach (var entry in header.Entries) + ArchiveSafety.ResolveEntryPath(targetDirectory, entry.Name, archiveName); + ArchiveSafety.EnsureRoom(header.Entries.Sum(e => e.Size), targetDirectory, archiveName); + + var dataStart = stream.Position; + foreach (var entry in header.Entries) + { + cancellationToken.ThrowIfCancellationRequested(); + var path = ArchiveSafety.ResolveEntryPath(targetDirectory, entry.Name, archiveName); + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + path = ArchiveFiles.UniquePath(path); + stream.Position = dataStart + entry.Offset; + var crc = new Crc32(); + using (var output = new CrcStream(new FileStream(path, FileMode.CreateNew, FileAccess.Write, FileShare.None, 1 << 16), crc)) + ArchiveFiles.CopyLimited(new BoundedStream(stream, entry.Size), output, entry.Size, cancellationToken); + if (new FileInfo(path).Length != entry.Size) + throw ArchiveSafety.Damaged(archiveName, $"{entry.Name} is cut off"); + // gmad -nocrc writes 0 for every file. + if (entry.Crc != 0 && crc.Value != entry.Crc) + throw ArchiveSafety.Damaged(archiveName, $"checksum error in {entry.Name}"); + if (header.Created is { } created) + File.SetLastWriteTimeUtc(path, created.UtcDateTime); + } + + var json = Path.Combine(targetDirectory, AddonJson); + if (!File.Exists(json)) + File.WriteAllText(json, AddonJsonText(header), new UTF8Encoding(false)); + } + + /// Reads the header and file list; leaves at the first file's content. + internal static Header ReadHeader(Stream stream, string archiveName) + { + try + { + Span ident = stackalloc byte[4]; + stream.ReadExactly(ident); + if (!ident.SequenceEqual(Ident)) + throw new InvalidDataException($"{archiveName} is not a Garry's Mod addon (GMA)."); + var reader = new BinaryReader(stream, Encoding.UTF8, leaveOpen: true); + var version = reader.ReadByte(); + if (version is 0 or > Version) + throw new InvalidDataException($"{archiveName} is a GMA of version {version}; Filee reads versions 1 to {Version}."); + reader.ReadUInt64(); // SteamID + var timestamp = reader.ReadUInt64(); + if (version > 1) + { + while (ReadString(stream, archiveName).Length > 0) + { + // required content, unused by Garry's Mod + } + } + var title = ReadString(stream, archiveName); + var description = ReadString(stream, archiveName); + var author = ReadString(stream, archiveName); + reader.ReadInt32(); // addon version + + var entries = new List(); + long offset = 0; + while (reader.ReadUInt32() != 0) + { + if (entries.Count >= ArchiveSafety.MaxEntries) + throw new InvalidDataException($"{archiveName} has more than {ArchiveSafety.MaxEntries:N0} files; Filee does not unpack it."); + var name = ReadString(stream, archiveName); + var size = reader.ReadInt64(); + var crc = reader.ReadUInt32(); + if (size < 0 || name.Length == 0) + throw ArchiveSafety.Damaged(archiveName, "the file list is broken"); + // Checked per file, so the sum never overflows: no file may be larger than what is left of the addon. + if (size > stream.Length - stream.Position - offset) + throw ArchiveSafety.Damaged(archiveName, $"{name} is larger than the addon"); + entries.Add(new Entry(name, size, crc, offset)); + offset += size; + } + // Every file's content must be there (the CRC of the whole addon follows it). + if (stream.Position + offset > stream.Length) + throw ArchiveSafety.Damaged(archiveName, "the addon is cut off"); + + var (text, type, tags) = ParseDescription(description); + DateTimeOffset? created = timestamp is > 0 and < 253402300800 ? DateTimeOffset.FromUnixTimeSeconds((long)timestamp) : null; + return new Header(title, text, type, tags, author, created, entries); + } + catch (EndOfStreamException) + { + throw ArchiveSafety.Damaged(archiveName, "the addon is cut off"); + } + } + + /// A NUL-terminated UTF-8 string. + private static string ReadString(Stream stream, string archiveName) + { + var bytes = new List(); + while (true) + { + var b = stream.ReadByte(); + if (b < 0) + throw new EndOfStreamException(); + if (b == 0) + return Encoding.UTF8.GetString([.. bytes]); + if (bytes.Count >= MaxNameBytes) + throw ArchiveSafety.Damaged(archiveName, "a name in the addon is too long"); + bytes.Add((byte)b); + } + } + + /// + /// The description, type and tags from gmad's JSON description; an old addon whose description is plain text keeps it + /// as it is. + /// + internal static (string Description, string? Type, IReadOnlyList Tags) ParseDescription(string description) + { + try + { + if (JsonNode.Parse(description) is JsonObject json) + { + var text = json["description"] is JsonValue d && d.TryGetValue(out var s) ? s : ""; + var type = json["type"] is JsonValue t && t.TryGetValue(out var typeName) ? typeName : null; + var tags = json["tags"] is JsonArray array + ? array.OfType().Select(v => v.TryGetValue(out var tag) ? tag : null).OfType().ToList() + : []; + return (text, type, tags); + } + } + catch (JsonException) + { + // plain text + } + return (description, null, []); + } + + /// The addon.json for an unpacked addon, as gmad reads it. + internal static string AddonJsonText(Header header) + { + var json = new JsonObject + { + ["title"] = header.Title, + ["type"] = header.Type ?? "", + ["tags"] = new JsonArray([.. header.Tags.Select(tag => (JsonNode?)JsonValue.Create(tag))]), + ["description"] = header.Description, + ["ignore"] = new JsonArray(), + }; + return json.ToJsonString(new JsonSerializerOptions { WriteIndented = true }) + "\n"; + } + + /// The top folders of an addon, the ones gmad's whitelist starts with. + private static readonly HashSet AddonFolders = new( + WhiteList.Select(rule => rule.TrimStart('!').Split('/')[0]), StringComparer.OrdinalIgnoreCase); + + /// + /// The folder whose content goes into an addon: , or the folder inside it that holds + /// addon.json or the addon's top folders (lua, materials, models …). An addon zipped as its folder, maybe with a + /// readme beside it ("My Addon/lua/…"), is found through folders that hold one subfolder, up to a few levels. + /// + public static string Root(string folder) + { + var current = folder; + for (var depth = 0; depth < 8; depth++) + { + if (File.Exists(Path.Combine(current, AddonJson)) + || Directory.EnumerateDirectories(current).Any(d => AddonFolders.Contains(Path.GetFileName(d)))) + return current; + var folders = Directory.GetDirectories(current); + if (folders.Length != 1) + break; + current = folders[0]; + } + return folder; + } + + /// The settings gmad takes from addon.json. + internal sealed record AddonSettings(string? Title, string Description, string Type, IReadOnlyList Tags, IReadOnlyList Ignore); + + /// Reads addon.json; a missing file gives the defaults (title from the folder, type "servercontent"). + internal static AddonSettings ReadSettings(string? jsonPath) + { + if (jsonPath is null || !File.Exists(jsonPath)) + return new AddonSettings(null, "Description", "servercontent", [], []); + JsonObject json; + try + { + json = JsonNode.Parse(File.ReadAllText(jsonPath), documentOptions: new JsonDocumentOptions + { + AllowTrailingCommas = true, + CommentHandling = JsonCommentHandling.Skip, + }) as JsonObject ?? throw new InvalidDataException("addon.json is not a JSON object."); + } + catch (JsonException ex) + { + throw new InvalidDataException($"addon.json can't be read: {ex.Message}"); + } + + static string? Text(JsonNode? node) => node is JsonValue v && v.TryGetValue(out var s) ? s : null; + static List Texts(JsonNode? node) => node is JsonArray array ? [.. array.Select(Text).OfType()] : []; + + var title = Text(json["title"]); + var type = Text(json["type"]); + return new AddonSettings( + string.IsNullOrWhiteSpace(title) ? null : title, + Text(json["description"]) ?? "Description", + string.IsNullOrWhiteSpace(type) ? "servercontent" : type.ToLowerInvariant(), + [.. Texts(json["tags"]).Where(t => t.Length > 0).Select(t => t.ToLowerInvariant())], + Texts(json["ignore"])); + } + + /// True when gmad's whitelist lets (lower case, "/" separated) into an addon. + internal static bool IsAllowed(string name) + { + var valid = false; + foreach (var rule in WhiteList) + { + if (rule.StartsWith('!')) + { + if (Wildcard(rule[1..], name)) + valid = false; + } + else if (!valid) + { + valid = Wildcard(rule, name); + } + } + return valid; + } + + /// True when one of gmad's own rules or an addon.json "ignore" wildcard leaves out. + internal static bool IsIgnored(string name, IReadOnlyList ignore) + { + var lower = name.ToLowerInvariant(); + if (lower == AddonJson || Wildcard("*thumbs.db", lower) || Wildcard("*desktop.ini", lower) || Wildcard(".git*", lower) + || name == ".DS_Store" || Wildcard("*/.DS_Store", name)) + return true; + return ignore.Any(pattern => Wildcard(pattern.Replace('\\', '/'), name)); + } + + /// Bootil's wildcard test: "*" matches any run of characters (slashes too), everything else itself. + internal static bool Wildcard(string pattern, string text) + { + int p = 0, t = 0, star = -1, mark = 0; + while (t < text.Length) + { + if (p < pattern.Length && pattern[p] == '*') + { + star = p++; + mark = t; + } + else if (p < pattern.Length && pattern[p] == text[t]) + { + p++; + t++; + } + else if (star >= 0) + { + p = star + 1; + t = ++mark; + } + else + { + return false; + } + } + while (p < pattern.Length && pattern[p] == '*') + p++; + return p == pattern.Length; + } + + /// + /// Writes an addon of (entry name relative to the addon folder, "/" separated, and the file + /// on disk). addon.json among them gives the title, type, tags and ignores; is the + /// title without one. Returns the names left out because Garry's Mod would not load them. + /// + public static IReadOnlyList Write(IReadOnlyList<(string EntryName, string Path)> files, string output, string fallbackTitle, + IProgress? progress, CancellationToken cancellationToken) + { + var json = files.FirstOrDefault(f => string.Equals(f.EntryName, AddonJson, StringComparison.OrdinalIgnoreCase)); + var settings = ReadSettings(json.Path); + var title = settings.Title ?? fallbackTitle; + + var packed = new List<(string Name, string Path, long Size)>(); + var skipped = new List(); + var names = new HashSet(StringComparer.Ordinal); + foreach (var (entryName, path) in files.OrderBy(f => f.EntryName, StringComparer.Ordinal)) + { + var name = entryName.Replace('\\', '/').TrimStart('/'); + if (IsIgnored(name, settings.Ignore)) + continue; + var lower = name.ToLowerInvariant(); + if (!IsAllowed(lower) || !names.Add(lower)) + { + skipped.Add(name); + continue; + } + packed.Add((lower, path, new FileInfo(path).Length)); + } + if (packed.Count == 0) + throw new InvalidDataException( + "None of these files can go into a Garry's Mod addon. An addon holds lua/, materials/, models/, sound/, maps/ and the like, as Garry's Mod loads them."); + + var description = new JsonObject + { + ["description"] = settings.Description, + ["type"] = settings.Type, + ["tags"] = new JsonArray([.. settings.Tags.Select(tag => (JsonNode?)JsonValue.Create(tag))]), + }.ToJsonString(new JsonSerializerOptions { WriteIndented = true }); + + var total = Math.Max(1, packed.Sum(f => f.Size)); + long done = 0; + var crc = new Crc32(); + using var file = new FileStream(output, FileMode.CreateNew, FileAccess.Write, FileShare.None, 1 << 16); + using var stream = new CrcStream(file, crc); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + writer.Write(Ident); + writer.Write(Version); + writer.Write(0UL); // SteamID + writer.Write((ulong)DateTimeOffset.UtcNow.ToUnixTimeSeconds()); + writer.Write((byte)0); // no required content + WriteString(writer, title); + WriteString(writer, description); + WriteString(writer, "Author Name"); // as gmad: unused + writer.Write(1); // addon version, unused + + var number = 0u; + foreach (var (name, path, size) in packed) + { + writer.Write(++number); + WriteString(writer, name); + writer.Write(size); + writer.Write(FileCrc(path, cancellationToken)); + } + writer.Write(0u); + } + foreach (var (_, path, size) in packed) + { + cancellationToken.ThrowIfCancellationRequested(); + using var source = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete, 1 << 16); + var copied = ArchiveFiles.CopyLimited(source, stream, size, cancellationToken); + if (copied != size) + throw new IOException($"{path} changed while it was packed."); + done += size; + progress?.Report((double)done / total); + } + // The CRC of everything written so far, which the game checks after a download. + Span tail = stackalloc byte[4]; + System.Buffers.Binary.BinaryPrimitives.WriteUInt32LittleEndian(tail, crc.Value); + file.Write(tail); + return skipped; + } + + private static void WriteString(BinaryWriter writer, string text) + { + writer.Write(Encoding.UTF8.GetBytes(text.Replace("\0", ""))); + writer.Write((byte)0); + } + + private static uint FileCrc(string path, CancellationToken cancellationToken) + { + var crc = new Crc32(); + var buffer = new byte[1 << 16]; + using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete, 1 << 16); + int read; + while ((read = stream.Read(buffer)) > 0) + { + cancellationToken.ThrowIfCancellationRequested(); + crc.Append(buffer.AsSpan(0, read)); + } + return crc.Value; + } +} diff --git a/src/Filee.Engines/Archives/Vpk.cs b/src/Filee.Engines/Archives/Vpk.cs new file mode 100644 index 000000000..3bfe3fb82 --- /dev/null +++ b/src/Filee.Engines/Archives/Vpk.cs @@ -0,0 +1,217 @@ +// Valve pak files (.vpk), the archives the Source engine keeps its games' files in (Half-Life 2, Counter-Strike: Source, +// Team Fortress 2, the content Garry's Mod mounts). Read by Filee; layout (little endian) as in Valve's vpklib and +// srctools: +// +// signature u32 0x55AA1234, version u32 (1 or 2), tree size u32 +// version 2: file data size, archive MD5 size, other MD5 size, signature size (u32 each; the sections are not read) +// the tree: extensions, in each its folders, in each its file names (NUL-terminated; an empty string ends each level, +// " " stands for no extension or no folder); per file a CRC-32 u32, preload size u16, archive index u16, +// offset u32, length u32 and 0xFFFF, then the preload bytes +// archive index 0x7FFF: the file's data follows the tree in the directory file itself; any other index n: the data +// is in "_nnn.vpk" next to "_dir.vpk" +// +// A file's content is its preload bytes followed by its data. The PS Vita's .vpk is a ZIP and goes to 7-Zip instead. + +using System.Buffers.Binary; +using System.Text; +using System.Text.RegularExpressions; + +namespace Filee.Engines.Archives; + +/// Unpacks Valve pak files. +internal static partial class Vpk +{ + private const uint Signature = 0x55AA1234; + + /// The archive index of data stored in the directory file after the tree. + private const ushort InDirectory = 0x7FFF; + + /// The longest name Filee reads from the tree (longer is a damaged file). + private const int MaxNameBytes = 4096; + + /// The largest tree Filee reads (Valve's largest, Team Fortress 2's, is about 20 MB). + private const uint MaxTreeBytes = 256 * 1024 * 1024; + + /// Data parts are numbered "_000" to "_999". + private const int MaxParts = 1000; + + /// One file of a pak. + internal sealed record Entry(string Name, uint Crc, byte[] Preload, int Archive, long Offset, long Length) + { + public long Size => Preload.Length + Length; + } + + /// A data part of a split pak ("pak01_003.vpk"), which holds no tree. + [GeneratedRegex(@"^(?.+)_(?\d{3})\.vpk$", RegexOptions.IgnoreCase)] + private static partial Regex DataPart(); + + /// True for a ZIP (a PS Vita package), which 7-Zip reads. + public static bool IsZip(string path) + { + Span start = stackalloc byte[4]; + using var stream = File.OpenRead(path); + return stream.ReadAtLeast(start, 4, throwOnEndOfStream: false) == 4 && start.SequenceEqual("PK\x03\x04"u8); + } + + /// Unpacks every file of the pak (its directory file) into . + public static void Extract(string archive, string targetDirectory, CancellationToken cancellationToken) + { + var archiveName = Path.GetFileName(archive); + using var directory = new FileStream(archive, FileMode.Open, FileAccess.Read, FileShare.Read, 1 << 16); + var (entries, dataStart) = ReadTree(directory, archiveName); + // Every name is checked before anything is written: one that leaves the folder stops the whole pak. + foreach (var entry in entries) + ArchiveSafety.ResolveEntryPath(targetDirectory, entry.Name, archiveName); + ArchiveSafety.EnsureRoom(entries.Sum(e => e.Size), targetDirectory, archiveName); + + var parts = new Dictionary(); + try + { + foreach (var entry in entries) + { + cancellationToken.ThrowIfCancellationRequested(); + var (stream, start) = entry.Archive == InDirectory + ? (directory, dataStart + entry.Offset) + : (Part(archive, entry.Archive, parts, archiveName), entry.Offset); + if (entry.Length > 0 && start + entry.Length > stream.Length) + throw ArchiveSafety.Damaged(archiveName, $"{entry.Name} is cut off"); + + var path = ArchiveSafety.ResolveEntryPath(targetDirectory, entry.Name, archiveName); + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + path = ArchiveFiles.UniquePath(path); + var crc = new Crc32(); + using (var output = new CrcStream(new FileStream(path, FileMode.CreateNew, FileAccess.Write, FileShare.None, 1 << 16), crc)) + { + output.Write(entry.Preload); + if (entry.Length > 0) + { + stream.Position = start; + ArchiveFiles.CopyLimited(new BoundedStream(stream, entry.Length), output, entry.Length, cancellationToken); + } + } + if (new FileInfo(path).Length != entry.Size) + throw ArchiveSafety.Damaged(archiveName, $"{entry.Name} is cut off"); + if (entry.Crc != 0 && crc.Value != entry.Crc) + throw ArchiveSafety.Damaged(archiveName, $"checksum error in {entry.Name}"); + } + } + finally + { + foreach (var part in parts.Values) + part.Dispose(); + } + } + + /// The data part of a split pak, opened once. + private static FileStream Part(string directoryFile, int index, Dictionary parts, string archiveName) + { + if (parts.TryGetValue(index, out var open)) + return open; + var folder = Path.GetDirectoryName(Path.GetFullPath(directoryFile))!; + var name = Path.GetFileNameWithoutExtension(directoryFile); + if (name.EndsWith("_dir", StringComparison.OrdinalIgnoreCase)) + name = name[..^4]; + var partName = $"{name}_{index:D3}.vpk"; + var path = Path.Combine(folder, partName); + if (!File.Exists(path)) + { + // Valve's tools write lower case; a copy may have changed it. + path = Directory.EnumerateFiles(folder).FirstOrDefault(f => string.Equals(Path.GetFileName(f), partName, StringComparison.OrdinalIgnoreCase)) + ?? throw new FileNotFoundException($"{archiveName} keeps files in {partName}, which is not next to it.", partName); + } + var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, 1 << 16); + parts[index] = stream; + return stream; + } + + /// Reads the header and the tree; returns the files and where the data in the directory file starts. + internal static (List Entries, long DataStart) ReadTree(Stream stream, string archiveName) + { + Span header = stackalloc byte[12]; + if (stream.ReadAtLeast(header, 12, throwOnEndOfStream: false) < 12 + || BinaryPrimitives.ReadUInt32LittleEndian(header) != Signature) + { + if (DataPart().Match(archiveName) is { Success: true } part) + throw new InvalidDataException( + $"{archiveName} is part {part.Groups["part"].Value} of a Valve pak; unpack {part.Groups["prefix"].Value}_dir.vpk, which lists its files."); + throw new InvalidDataException($"{archiveName} is not a Valve pak (VPK)."); + } + var version = BinaryPrimitives.ReadUInt32LittleEndian(header[4..]); + var treeSize = BinaryPrimitives.ReadUInt32LittleEndian(header[8..]); + if (version is not (1 or 2)) + throw new InvalidDataException($"{archiveName} is a VPK of version {version}; Filee reads versions 1 and 2."); + var headerSize = version == 2 ? 28 : 12; + var dataStart = (long)headerSize + treeSize; + if (treeSize > MaxTreeBytes) + throw new InvalidDataException($"{archiveName} lists more files than Filee unpacks from one pak."); + if (dataStart > stream.Length) + throw ArchiveSafety.Damaged(archiveName, "the pak is cut off"); + stream.Position = headerSize; + + try + { + var reader = new BinaryReader(stream, Encoding.UTF8, leaveOpen: true); + var entries = new List(); + Span fields = stackalloc byte[18]; + for (string extension; (extension = ReadString(stream, archiveName)).Length > 0;) + { + for (string folder; (folder = ReadString(stream, archiveName)).Length > 0;) + { + for (string file; (file = ReadString(stream, archiveName)).Length > 0;) + { + if (entries.Count >= ArchiveSafety.MaxEntries) + throw new InvalidDataException($"{archiveName} has more than {ArchiveSafety.MaxEntries:N0} files; Filee does not unpack it."); + stream.ReadExactly(fields); + var crc = BinaryPrimitives.ReadUInt32LittleEndian(fields); + var preloadSize = BinaryPrimitives.ReadUInt16LittleEndian(fields[4..]); + var archive = BinaryPrimitives.ReadUInt16LittleEndian(fields[6..]); + var offset = BinaryPrimitives.ReadUInt32LittleEndian(fields[8..]); + var length = BinaryPrimitives.ReadUInt32LittleEndian(fields[12..]); + if (BinaryPrimitives.ReadUInt16LittleEndian(fields[16..]) != 0xFFFF + || (archive != InDirectory && archive >= MaxParts)) + throw ArchiveSafety.Damaged(archiveName, "the file list is broken"); + // The preload bytes are part of the tree: checked against it before they are read. + if (stream.Position + preloadSize > dataStart) + throw ArchiveSafety.Damaged(archiveName, "the file list is broken"); + var preload = reader.ReadBytes(preloadSize); + if (preload.Length != preloadSize) + throw new EndOfStreamException(); + entries.Add(new Entry(Join(folder, file, extension), crc, preload, archive, offset, length)); + } + } + } + if (stream.Position > dataStart) + throw ArchiveSafety.Damaged(archiveName, "the file list is broken"); + return (entries, dataStart); + } + catch (EndOfStreamException) + { + throw ArchiveSafety.Damaged(archiveName, "the pak is cut off"); + } + } + + /// "folder/name.ext"; " " (Valve's tools) or "" stand for no folder or no extension. + private static string Join(string folder, string name, string extension) + { + folder = folder.Trim() is "" ? "" : folder.Trim().TrimEnd('/') + "/"; + extension = extension.Trim() is "" ? "" : "." + extension.Trim(); + return folder + name + extension; + } + + /// A NUL-terminated string (Valve writes ASCII; UTF-8 reads it too). + private static string ReadString(Stream stream, string archiveName) + { + var bytes = new List(); + while (true) + { + var b = stream.ReadByte(); + if (b < 0) + throw new EndOfStreamException(); + if (b == 0) + return Encoding.UTF8.GetString([.. bytes]); + if (bytes.Count >= MaxNameBytes) + throw ArchiveSafety.Damaged(archiveName, "a name in the pak is too long"); + bytes.Add((byte)b); + } + } +} diff --git a/src/Filee.Engines/Models3D/ModelConverter.cs b/src/Filee.Engines/Models3D/ModelConverter.cs index 03068c326..5139dc9b0 100644 --- a/src/Filee.Engines/Models3D/ModelConverter.cs +++ b/src/Filee.Engines/Models3D/ModelConverter.cs @@ -1,6 +1,7 @@ // 3D models in managed code only (SharpGLTF for glTF, Filee's own readers and writers for the rest): GLB, glTF, STL, -// OBJ, 3MF and PLY into each other through one in-memory model (Model3D), FBX (read only) into any of them, and an -// isometric preview picture. +// OBJ, 3MF and PLY into each other through one in-memory model (Model3D), FBX and Source engine models (MDL, with their +// textures; read only) into any of them, studiomdl data (SMD) to and from them (from an MDL with its bones and weights), +// and an isometric preview picture. // See docs/ENGINES.md, "3D models", for units, axes and what each format keeps. using Filee.Core.Conversion; @@ -19,8 +20,8 @@ public sealed class ModelConverter(ILogger? logger = null) : IConverter, IImageT /// The 3D formats, each readable and writable. public static readonly string[] Formats = ["glb", "gltf", "stl", "obj", "3mf", "ply"]; - /// Every 3D format that can be read: and FBX. - public static readonly string[] InputFormats = [.. Formats, "fbx"]; + /// Every 3D format that can be read: , FBX, Source engine models (MDL) and SMD. + public static readonly string[] InputFormats = [.. Formats, "fbx", "mdl", "smd"]; /// Long edge of the preview picture before the preset's resize. internal const int PreviewSize = 1024; @@ -39,15 +40,20 @@ public sealed class ModelConverter(ILogger? logger = null) : IConverter, IImageT public IReadOnlyList Edges { get; } = [ .. from source in InputFormats from target in Formats where source != target select new ConversionEdge(source, target), + // SMD, the source studiomdl compiles: from an MDL with its skeleton, from the others as a static prop. + .. from source in InputFormats where source != "smd" select new ConversionEdge(source, "smd"), .. from source in InputFormats from target in ImageEncoder.Writable select ImageEncoder.Edge(source, target), ]; - public EngineStatus GetStatus() => EngineStatus.Available("GLB, glTF, STL, OBJ, 3MF, PLY ↔ each other; FBX → each of them", + public EngineStatus GetStatus() => EngineStatus.Available("GLB, glTF, STL, OBJ, 3MF, PLY, SMD ↔ each other; FBX, MDL → each of them", $"{EngineVersions.BuiltIn} · {EngineVersions.Library("SharpGLTF", typeof(SharpGLTF.Schema2.ModelRoot))}"); public Task> ConvertAsync(ConversionStep step, IProgress? progress, CancellationToken cancellationToken) => Task.Run>(() => { + // A Source model keeps its skeleton and weights in an SMD, which Filee's static model has no place for. + if (step.From == "mdl" && step.To == "smd") + return WriteSkeletalSmd(step, cancellationToken); var reading = progress is null ? null : new SubProgress(progress, 0, 0.6); var model = Read(step.InputPath, step.From, reading, cancellationToken); _logger.LogDebug("{File}: {Parts} parts, {Vertices} vertices, {Triangles} triangles, {Materials} materials", @@ -80,6 +86,8 @@ internal static Model3D Read(string path, string format, IProgress? prog "3mf" => ThreeMfFormat.Read(path, progress, cancellationToken), "glb" or "gltf" => GltfFormat.Read(path, progress, cancellationToken), "fbx" => FbxFormat.Read(path, progress, cancellationToken), + "mdl" => SourceModel.Load(path, cancellationToken).ToModel3D(path, cancellationToken), + "smd" => SmdFormat.Read(path, progress, cancellationToken), _ => throw new NotSupportedException($"Unknown 3D format '{format}'."), }; model.Validate(); @@ -97,7 +105,7 @@ internal static void Write(Model3D model, string format, IOutputAllocator output if (output.Allocate(format) is not { } path) return; outputs.Add(path); - if (!Formats.Contains(format)) + if (!Formats.Contains(format) && format != "smd") { WriteImage(model, format, imageOptions, path, cancellationToken); return; @@ -114,6 +122,9 @@ internal static void Write(Model3D model, string format, IOutputAllocator output case "3mf": ThreeMfFormat.Write(model, stream, cancellationToken); break; + case "smd": + SmdFormat.Write(model, stream, cancellationToken); + break; case "glb": case "gltf": GltfFormat.Write(model, stream, binary: format == "glb", cancellationToken); @@ -121,6 +132,25 @@ internal static void Write(Model3D model, string format, IOutputAllocator output } } + /// A Source model as a reference SMD with its bones, bind pose and weights. + private static IReadOnlyList WriteSkeletalSmd(ConversionStep step, CancellationToken cancellationToken) + { + var model = SourceModel.Load(step.InputPath, cancellationToken); + if (step.Output.Allocate("smd") is not { } path) + return []; + try + { + using var stream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.None, 1 << 16); + SmdFormat.WriteSkeletal(model, stream, cancellationToken); + } + catch + { + TryDelete(path); + throw; + } + return [path]; + } + /// The .obj, the .mtl next to it (when the model has materials) and the textures next to that. private static void WriteObj(Model3D model, IOutputAllocator output, List outputs, CancellationToken cancellationToken) { diff --git a/src/Filee.Engines/Models3D/SmdFormat.cs b/src/Filee.Engines/Models3D/SmdFormat.cs new file mode 100644 index 000000000..24dbc9e67 --- /dev/null +++ b/src/Filee.Engines/Models3D/SmdFormat.cs @@ -0,0 +1,207 @@ +// Studiomdl data (.smd), the text format Valve's studiomdl compiles Source engine models from (Blender Source Tools, +// Crowbar): +// +// version 1 +// nodes one line per bone: index "name" parent (−1 for a root) +// skeleton / time 0 one line per bone: index, position x y z, rotation x y z (radians), relative to its parent +// triangles per triangle a material line, then three vertex lines: +// parent bone, position x y z, normal x y z, u v [, link count, (bone weight) per link] +// end after each section +// +// Positions are in model space, Z up, in inches (Source's unit), triangles counter-clockwise, UVs from the bottom left. +// Reading takes the triangles of a reference SMD as a static model (an animation SMD has none). Writing from another +// model gives a static prop: one root bone, every vertex weighted to it. Writing from a Source model (MDL) keeps its +// bones, bind pose and weights, as a reference mesh Crowbar would decompile. + +using System.Globalization; +using System.Numerics; +using System.Text; + +namespace Filee.Engines.Models3D; + +/// Reads and writes studiomdl data files. +internal static class SmdFormat +{ + /// The most triangles Filee reads from one file (about 1 GB of text). + private const int MaxTriangles = 10_000_000; + + private static readonly CultureInfo Invariant = CultureInfo.InvariantCulture; + + /// The triangles of a reference SMD, one part per material. + public static Model3D Read(string path, IProgress? progress, CancellationToken cancellationToken) + { + var model = new Model3D(); + var parts = new Dictionary(StringComparer.Ordinal); + var length = Math.Max(1, new FileInfo(path).Length); + using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, 1 << 16); + using var reader = new StreamReader(stream, Encoding.UTF8, detectEncodingFromByteOrderMarks: true); + var lineNumber = 0; + string? Next() + { + while (reader.ReadLine() is { } line) + { + lineNumber++; + var text = line.Trim(); + if (text.Length > 0 && !text.StartsWith("//", StringComparison.Ordinal)) + return text; + } + return null; + } + + var triangles = 0; + var corners = new int[3]; + string? section = null; + while (Next() is { } line) + { + if (section is null) + { + section = line.Split((char[]?)null, 2, StringSplitOptions.RemoveEmptyEntries)[0].ToLowerInvariant(); + if (section is "version") + section = null; + continue; + } + if (line.Equals("end", StringComparison.OrdinalIgnoreCase)) + { + section = null; + continue; + } + if (section != "triangles") + continue; // nodes, skeleton, vertex animation: not part of the static mesh + + // A material line, then its triangle's three vertices. + cancellationToken.ThrowIfCancellationRequested(); + if (++triangles > MaxTriangles) + throw new InvalidDataException($"The SMD has more than {MaxTriangles:N0} triangles."); + var material = line; + if (!parts.TryGetValue(material, out var part)) + { + model.Materials.Add(new Material3D { Name = Path.GetFileNameWithoutExtension(material) }); + part = new MeshPart { Name = Path.GetFileNameWithoutExtension(material), Material = model.Materials.Count - 1, Normals = [], TexCoords = [] }; + parts[material] = part; + model.Parts.Add(part); + } + for (var k = 0; k < 3; k++) + { + var vertex = Next() ?? throw new InvalidDataException($"The SMD ends inside a triangle (line {lineNumber})."); + var fields = vertex.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries); + if (fields.Length < 9) + throw new InvalidDataException($"Line {lineNumber} of the SMD is not a triangle's vertex."); + var position = new Vector3(Number(fields[1], lineNumber), Number(fields[2], lineNumber), Number(fields[3], lineNumber)); + var normal = new Vector3(Number(fields[4], lineNumber), Number(fields[5], lineNumber), Number(fields[6], lineNumber)); + var uv = new Vector2(Number(fields[7], lineNumber), Number(fields[8], lineNumber)); + corners[k] = part.AddVertex(MeshMath.ToYUp(position) * SourceModel.MillimetresPerUnit, MeshMath.ToYUp(normal), uv); + } + part.AddTriangle(corners[0], corners[1], corners[2]); + progress?.Report(Math.Min(1, (double)stream.Position / length)); + } + return model; + } + + private static float Number(string text, int line) => + float.TryParse(text, NumberStyles.Float, Invariant, out var value) && float.IsFinite(value) + ? value + : throw new InvalidDataException($"Line {line} of the SMD has \"{text}\" where a number belongs."); + + /// A static prop: one root bone at the origin, every vertex weighted to it. + public static void Write(Model3D model, Stream stream, CancellationToken cancellationToken) + { + using var writer = new StreamWriter(stream, new UTF8Encoding(false), 1 << 16, leaveOpen: true) { NewLine = "\n" }; + writer.WriteLine("version 1"); + writer.WriteLine("nodes"); + writer.WriteLine("0 \"root\" -1"); + writer.WriteLine("end"); + writer.WriteLine("skeleton"); + writer.WriteLine("time 0"); + writer.WriteLine("0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000"); + writer.WriteLine("end"); + writer.WriteLine("triangles"); + foreach (var part in model.Parts) + { + if (part.Normals is null) + part.ComputeNormals(); + var material = MaterialName(model.MaterialOf(part), part.Name); + var indices = part.Indices; + for (var t = 0; t + 2 < indices.Count; t += 3) + { + if (t % 3000 == 0) + cancellationToken.ThrowIfCancellationRequested(); + writer.WriteLine(material); + for (var k = 0; k < 3; k++) + { + var v = indices[t + k]; + var position = MeshMath.ToZUp(part.Position(v)) / SourceModel.MillimetresPerUnit; + WriteVertex(writer, 0, position, MeshMath.ToZUp(part.Normal(v)), part.TexCoord(v)); + writer.WriteLine(" 1 0 1.000000"); + } + } + } + writer.WriteLine("end"); + } + + /// A Source model as a reference SMD: its bones in their bind pose, its triangles with their weights. + public static void WriteSkeletal(SourceModel model, Stream stream, CancellationToken cancellationToken) + { + using var writer = new StreamWriter(stream, new UTF8Encoding(false), 1 << 16, leaveOpen: true) { NewLine = "\n" }; + writer.WriteLine("version 1"); + writer.WriteLine("nodes"); + var bones = model.Bones.Count > 0 ? model.Bones : [new SourceModel.Bone("root", -1, Vector3.Zero, Vector3.Zero)]; + for (var i = 0; i < bones.Count; i++) + writer.WriteLine($"{i} \"{bones[i].Name.Replace('"', '\'')}\" {bones[i].Parent}"); + writer.WriteLine("end"); + writer.WriteLine("skeleton"); + writer.WriteLine("time 0"); + for (var i = 0; i < bones.Count; i++) + { + var (p, r) = (bones[i].Position, bones[i].Rotation); + writer.WriteLine(string.Create(Invariant, $"{i} {p.X:F6} {p.Y:F6} {p.Z:F6} {r.X:F6} {r.Y:F6} {r.Z:F6}")); + } + writer.WriteLine("end"); + writer.WriteLine("triangles"); + foreach (var mesh in model.Meshes) + { + cancellationToken.ThrowIfCancellationRequested(); + var material = Clean(mesh.Material); + // Source's triangles are clockwise; an SMD's counter-clockwise, as Crowbar writes them. + for (var t = 0; t + 2 < mesh.Triangles.Count; t += 3) + { + writer.WriteLine(material); + foreach (var index in new[] { mesh.Triangles[t + 2], mesh.Triangles[t + 1], mesh.Triangles[t] }) + { + var v = model.Vertices[index]; + var bone = Math.Clamp(v.Bone0, 0, bones.Count - 1); + WriteVertex(writer, bone, v.Position, v.Normal, new Vector2(v.Uv.X, 1 - v.Uv.Y)); + var links = new StringBuilder(); + var count = 0; + foreach (var (b, w) in new[] { (v.Bone0, v.Weight0), (v.Bone1, v.Weight1), (v.Bone2, v.Weight2) }.Take(v.BoneCount)) + { + if (b < 0 || b >= bones.Count) + continue; + links.Append(Invariant, $" {b} {w:F6}"); + count++; + } + writer.WriteLine(count > 0 ? $" {count}{links}" : $" 1 {bone} 1.000000"); + } + } + } + writer.WriteLine("end"); + } + + private static void WriteVertex(StreamWriter writer, int bone, Vector3 p, Vector3 n, Vector2 uv) => + writer.Write(string.Create(Invariant, $"{bone} {p.X:F6} {p.Y:F6} {p.Z:F6} {n.X:F6} {n.Y:F6} {n.Z:F6} {uv.X:F6} {uv.Y:F6}")); + + /// The material line: studiomdl looks the name up as a .vmt, so it is the material's or texture's name. + private static string MaterialName(Material3D? material, string partName) + { + var name = material?.Name is { Length: > 0 } named ? named + : material?.BaseColorTexture?.Name is { Length: > 0 } texture ? texture + : partName; + return Clean(name); + } + + /// One word without spaces or quotes; "default" when nothing is left. + private static string Clean(string name) + { + var clean = new string([.. name.Trim().Select(c => char.IsWhiteSpace(c) || c == '"' ? '_' : c)]); + return clean.Length == 0 ? "default" : clean; + } +} diff --git a/src/Filee.Engines/Models3D/SourceModel.cs b/src/Filee.Engines/Models3D/SourceModel.cs new file mode 100644 index 000000000..4479e33f9 --- /dev/null +++ b/src/Filee.Engines/Models3D/SourceModel.cs @@ -0,0 +1,582 @@ +// Source engine models (.mdl, versions 44 to 49: Garry's Mod, Counter-Strike: Source, Team Fortress 2, Half-Life 2), read +// by Filee with the two files studiomdl writes beside them: the vertices (.vvd) and the triangles (.dx90.vtx, else .vtx, +// .dx80.vtx or .sw.vtx). Layouts (little endian) as in Valve's studio.h / optimize.h and the readers of SourceIO and +// Crowbar: +// +// .mdl studiohdr_t (408 bytes): bones (216 bytes each: name, parent, position, rotation), textures (64 bytes each: +// name), texture folders ($cdmaterials), the skin table (shorts: family × reference), body parts (16 bytes: name, +// models) → models (148 bytes: name, meshes, first vertex) → meshes (116 bytes: material, vertex offset) +// .vvd "IDSV", version 4, the checksum of its .mdl, vertices per LOD, fixups (which runs of vertices LOD 0 uses); +// 48 bytes a vertex: three bone weights, three bones, bone count, position, normal, UV +// .vtx version 7: body parts → models → LODs → meshes (9 bytes) → strip groups (25 bytes; 33 in later branches) +// → their vertices (9 bytes: the index of the mesh's vertex) and triangle lists (or strips) of u16 +// +// The model is read in its bind pose with the default body groups (the first model of each body part), the first skin +// and LOD 0; animations, flexes and physics (.phy) are not read. Source is Z up, one unit an inch, triangles clockwise, +// UVs from the top left: Filee turns that into its own conventions (Y up, millimetres, counter-clockwise, bottom left). +// The textures are found as Garry's Mod finds them: the model's material (a .vmt) in a "materials" folder above the +// model, its $basetexture (a .vtf) decoded by the VTF reader. + +using System.Buffers.Binary; +using System.Numerics; +using System.Text; +using Filee.Engines.Magick; +using ImageMagick; + +namespace Filee.Engines.Models3D; + +/// A Source engine model as read from its .mdl, .vvd and .vtx. +internal sealed class SourceModel +{ + /// One inch, Source's unit, in Filee's millimetres. + public const float MillimetresPerUnit = 25.4f; + + /// The largest .mdl, .vvd or .vtx Filee reads (a damaged or made-up file otherwise). + private const long MaxFileSize = 256L * 1024 * 1024; + + private const int MaxBones = 1024; + private const int MaxTextures = 4096; + private const int MaxBodyParts = 512; + private const int MaxMeshes = 16384; + private const int MaxVertices = 4_000_000; + + /// A bone in its bind pose, relative to its parent (Source's axes and units). + internal readonly record struct Bone(string Name, int Parent, Vector3 Position, Vector3 Rotation); + + /// A vertex in Source's axes and units, UV from the top left, with up to three bone weights. + internal readonly record struct Vertex(Vector3 Position, Vector3 Normal, Vector2 Uv, int BoneCount, + int Bone0, int Bone1, int Bone2, float Weight0, float Weight1, float Weight2); + + /// One mesh: its part name, material and triangles (indices into , Source's winding). + internal sealed record Mesh(string Part, string Material, List Triangles); + + public string Name { get; private init; } = ""; + public List Bones { get; } = []; + public List Vertices { get; } = []; + public List Meshes { get; } = []; + + /// The folders the model's materials are looked up in ($cdmaterials, "/" separated, without the end slash). + public List MaterialFolders { get; } = []; + + /// Reads the model at with its .vvd and .vtx. + public static SourceModel Load(string mdlPath, CancellationToken cancellationToken) + { + var mdl = new Bytes(ReadFile(mdlPath), Path.GetFileName(mdlPath)); + var basePath = Path.Combine(Path.GetDirectoryName(Path.GetFullPath(mdlPath))!, Path.GetFileNameWithoutExtension(mdlPath)); + var vvdPath = Companion(basePath, [".vvd"]) + ?? throw new FileNotFoundException($"{Path.GetFileName(mdlPath)} needs its vertex file {Path.GetFileName(basePath)}.vvd next to it."); + var vtxPath = Companion(basePath, [".dx90.vtx", ".vtx", ".dx80.vtx", ".sw.vtx"]) + ?? throw new FileNotFoundException($"{Path.GetFileName(mdlPath)} needs its triangle file {Path.GetFileName(basePath)}.dx90.vtx next to it."); + var vvd = new Bytes(ReadFile(vvdPath), Path.GetFileName(vvdPath)); + var vtx = new Bytes(ReadFile(vtxPath), Path.GetFileName(vtxPath)); + + if (!mdl.Data.AsSpan().StartsWith("IDST"u8)) + throw new InvalidDataException($"{mdl.File} is not a Source engine model (IDST)."); + var version = mdl.I32(4); + if (version is < 44 or > 49) + throw new InvalidDataException($"{mdl.File} is a model of version {version}; Filee reads Source engine models of versions 44 to 49."); + var checksum = mdl.I32(8); + var model = new SourceModel { Name = mdl.String(12, 64) }; + + ReadBones(mdl, model); + var textures = ReadTextures(mdl, model); + ReadVertices(vvd, checksum, model); + cancellationToken.ThrowIfCancellationRequested(); + if (!vtx.Data.AsSpan().StartsWith((ReadOnlySpan)[7, 0, 0, 0])) + throw new InvalidDataException($"{vtx.File} is not a triangle file of version 7."); + if (vtx.I32(16) != checksum) + throw new InvalidDataException($"{vtx.File} belongs to another build of {mdl.File}: its checksum differs."); + // Strip groups and strips grew by 8 bytes in later branches (CS:GO); the layout whose indices all fit is used. + try + { + ReadMeshes(mdl, vtx, textures, model, extended: false, cancellationToken); + } + catch (InvalidDataException) + { + model.Meshes.Clear(); + ReadMeshes(mdl, vtx, textures, model, extended: true, cancellationToken); + } + if (model.Meshes.Count == 0) + throw new InvalidDataException($"{mdl.File} has no triangles in its default body groups."); + return model; + } + + private static byte[] ReadFile(string path) + { + var length = new FileInfo(path).Length; + if (length > MaxFileSize) + throw new InvalidDataException($"{Path.GetFileName(path)} is too large to be a Source engine model file."); + return File.ReadAllBytes(path); + } + + /// The first of " + suffix" that exists, letter case aside. + private static string? Companion(string basePath, string[] suffixes) + { + var folder = Path.GetDirectoryName(basePath)!; + var name = Path.GetFileName(basePath); + var files = Directory.EnumerateFiles(folder).ToList(); + foreach (var suffix in suffixes) + { + var match = files.FirstOrDefault(f => string.Equals(Path.GetFileName(f), name + suffix, StringComparison.OrdinalIgnoreCase)); + if (match is not null) + return match; + } + return null; + } + + private static void ReadBones(Bytes mdl, SourceModel model) + { + var count = mdl.Count(156, MaxBones, "bones"); + var start = mdl.I32(160); + for (var i = 0; i < count; i++) + { + var at = start + i * 216; + var parent = mdl.I32(at + 4); + model.Bones.Add(new Bone(mdl.String(at + mdl.I32(at)), parent is >= -1 && parent < count ? parent : -1, + mdl.Vector(at + 32), mdl.Vector(at + 60))); + } + } + + /// The texture name of every skin reference of the first skin family. + private static List ReadTextures(Bytes mdl, SourceModel model) + { + var count = mdl.Count(204, MaxTextures, "textures"); + var start = mdl.I32(208); + var names = new List(count); + for (var i = 0; i < count; i++) + { + var at = start + i * 64; + names.Add(mdl.String(at + mdl.I32(at)).Replace('\\', '/')); + } + + var folders = mdl.Count(212, 256, "texture folders"); + var folderTable = mdl.I32(216); + for (var i = 0; i < folders; i++) + model.MaterialFolders.Add(mdl.String(mdl.I32(folderTable + i * 4)).Replace('\\', '/').Trim('/')); + + // The skin table's first row maps each mesh's material reference to a texture. + var references = mdl.Count(220, MaxTextures, "skin references"); + var families = mdl.Count(224, MaxTextures, "skins"); + var skinTable = mdl.I32(228); + var skin = new List(references); + for (var i = 0; i < references; i++) + { + var texture = families > 0 ? mdl.I16(skinTable + i * 2) : i; + skin.Add(texture >= 0 && texture < names.Count ? names[texture] : i < names.Count ? names[i] : ""); + } + return skin.Count > 0 ? skin : names; + } + + private static void ReadVertices(Bytes vvd, int checksum, SourceModel model) + { + if (!vvd.Data.AsSpan().StartsWith("IDSV"u8)) + throw new InvalidDataException($"{vvd.File} is not a Source engine vertex file (IDSV)."); + if (vvd.I32(8) != checksum) + throw new InvalidDataException($"{vvd.File} belongs to another build of the model: its checksum differs."); + var lod0 = vvd.Count(16, MaxVertices, "vertices"); + var fixups = vvd.Count(48, MaxVertices, "fixups"); + var fixupTable = vvd.I32(52); + var vertexData = vvd.I32(56); + // The tables are checked against the file before a single vertex is read or added. + bool Fits(long offset, long count, int size) => offset >= 0 && count >= 0 && offset + count * size <= vvd.Data.Length; + if (!Fits(vertexData, lod0, 48) || !Fits(fixupTable, fixups, 12)) + throw vvd.Damaged("the vertex table"); + + void Add(int index) + { + var at = vertexData + index * 48; + model.Vertices.Add(new Vertex(vvd.Vector(at + 16), vvd.Vector(at + 28), new Vector2(vvd.F32(at + 40), vvd.F32(at + 44)), + Math.Clamp((int)vvd.Byte(at + 15), 1, 3), vvd.Byte(at + 12), vvd.Byte(at + 13), vvd.Byte(at + 14), + vvd.F32(at), vvd.F32(at + 4), vvd.F32(at + 8))); + } + + if (fixups == 0) + { + for (var i = 0; i < lod0; i++) + Add(i); + return; + } + // With fixups, LOD 0 is every run of vertices whose LOD is 0 or above, in order. + for (var f = 0; f < fixups; f++) + { + var at = fixupTable + f * 12; + var lod = vvd.I32(at); + var source = vvd.I32(at + 4); + var length = vvd.I32(at + 8); + if (lod < 0) + continue; + if (source < 0 || length < 0 || model.Vertices.Count + length > MaxVertices || !Fits(vertexData, (long)source + length, 48)) + throw vvd.Damaged("a vertex fixup"); + for (var i = 0; i < length; i++) + Add(source + i); + } + } + + private static void ReadMeshes(Bytes mdl, Bytes vtx, List materials, SourceModel model, bool extended, CancellationToken cancellationToken) + { + var stripGroupSize = extended ? 33 : 25; + var stripSize = extended ? 35 : 27; + var bodyParts = Math.Min(mdl.Count(232, MaxBodyParts, "body parts"), vtx.Count(28, MaxBodyParts, "body parts")); + var mdlBodyParts = mdl.I32(236); + var vtxBodyParts = vtx.I32(32); + for (var b = 0; b < bodyParts; b++) + { + var mdlPart = mdlBodyParts + b * 16; + var vtxPart = vtxBodyParts + b * 8; + if (mdl.I32(mdlPart + 4) <= 0 || vtx.I32(vtxPart) <= 0) + continue; + // The default body group: the first model of the part. + var mdlModel = mdlPart + mdl.I32(mdlPart + 12); + var vtxModel = vtxPart + vtx.I32(vtxPart + 4); + var modelName = mdl.String(mdlModel, 64); + var firstVertex = mdl.I32(mdlModel + 84) / 48; + var meshes = mdl.Count(mdlModel + 72, MaxMeshes, "meshes"); + var mdlMeshes = mdlModel + mdl.I32(mdlModel + 76); + if (vtx.I32(vtxModel) <= 0) + continue; + var lod = vtxModel + vtx.I32(vtxModel + 4); // LOD 0 + meshes = Math.Min(meshes, vtx.Count(lod, MaxMeshes, "meshes")); + var vtxMeshes = lod + vtx.I32(lod + 4); + + for (var m = 0; m < meshes; m++) + { + cancellationToken.ThrowIfCancellationRequested(); + var mdlMesh = mdlMeshes + m * 116; + var material = mdl.I32(mdlMesh); + var meshFirst = firstVertex + mdl.I32(mdlMesh + 12); + var vtxMesh = vtxMeshes + m * 9; + var groups = vtx.Count(vtxMesh, MaxMeshes, "strip groups"); + var groupTable = vtxMesh + vtx.I32(vtxMesh + 4); + var triangles = new List(); + for (var g = 0; g < groups; g++) + { + var group = groupTable + g * stripGroupSize; + var vertexCount = vtx.Count(group, MaxVertices, "vertices"); + var vertexTable = group + vtx.I32(group + 4); + var indexCount = vtx.Count(group + 8, MaxVertices * 3, "indices"); + var indexTable = group + vtx.I32(group + 12); + var strips = vtx.Count(group + 16, MaxMeshes, "strips"); + var stripTable = group + vtx.I32(group + 20); + + int MeshVertex(int index) + { + if ((uint)index >= (uint)vertexCount) + throw vtx.Damaged("a triangle"); + var vertex = meshFirst + vtx.U16(vertexTable + index * 9 + 4); + if ((uint)vertex >= (uint)model.Vertices.Count) + throw vtx.Damaged("a triangle"); + return vertex; + } + + var anyStrip = false; + for (var s = 0; s < strips; s++) + { + var strip = stripTable + s * stripSize; + if ((vtx.Byte(strip + 18) & 0x02) == 0) + continue; + // A triangle strip: every next index makes a triangle with the two before it, alternating. + anyStrip = true; + var first = vtx.I32(strip + 4); + var count = vtx.I32(strip); + if (first < 0 || count < 0 || first + count > indexCount) + throw vtx.Damaged("a triangle strip"); + for (var i = 2; i < count; i++) + { + var a = MeshVertex(vtx.U16(indexTable + (first + i - 2) * 2)); + var c = MeshVertex(vtx.U16(indexTable + (first + i - 1) * 2)); + var d = MeshVertex(vtx.U16(indexTable + (first + i) * 2)); + if (a == c || c == d || a == d) + continue; + if (i % 2 == 0) + triangles.AddRange([a, c, d]); + else + triangles.AddRange([c, a, d]); + } + } + if (anyStrip) + continue; + if (indexCount % 3 != 0) + throw vtx.Damaged("a triangle list"); + for (var i = 0; i < indexCount; i++) + triangles.Add(MeshVertex(vtx.U16(indexTable + i * 2))); + } + if (triangles.Count == 0) + continue; + var name = material >= 0 && material < materials.Count ? materials[material] : $"material{material}"; + model.Meshes.Add(new Mesh(modelName, name, triangles)); + } + } + } + + /// + /// The model in Filee's conventions: one part per mesh, its textures looked up in the "materials" folder above the + /// model (none when there is no such folder or the game's own textures are meant). + /// + public Model3D ToModel3D(string mdlPath, CancellationToken cancellationToken) + { + var result = new Model3D(); + var textures = new SourceMaterials(mdlPath, MaterialFolders); + var materialIndex = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var mesh in Meshes) + { + cancellationToken.ThrowIfCancellationRequested(); + if (!materialIndex.TryGetValue(mesh.Material, out var index)) + { + index = result.Materials.Count; + materialIndex[mesh.Material] = index; + var (texture, doubleSided) = textures.Find(mesh.Material); + result.Materials.Add(new Material3D + { + Name = Path.GetFileName(mesh.Material), + BaseColorTexture = texture, + DoubleSided = doubleSided, + }); + } + + var part = new MeshPart { Name = mesh.Part, Material = index, Normals = [], TexCoords = [] }; + var local = new Dictionary(); + int Local(int vertex) + { + if (local.TryGetValue(vertex, out var at)) + return at; + var v = Vertices[vertex]; + at = part.AddVertex(MeshMath.ToYUp(v.Position) * MillimetresPerUnit, MeshMath.ToYUp(v.Normal), new Vector2(v.Uv.X, 1 - v.Uv.Y)); + local[vertex] = at; + return at; + } + // Source's triangles are clockwise: reversed, they face the same way counter-clockwise. + for (var t = 0; t + 2 < mesh.Triangles.Count; t += 3) + part.AddTriangle(Local(mesh.Triangles[t + 2]), Local(mesh.Triangles[t + 1]), Local(mesh.Triangles[t])); + result.Parts.Add(part); + } + return result; + } + + /// A model file's bytes with bounds-checked little-endian reads. + private sealed class Bytes(byte[] data, string file) + { + public byte[] Data { get; } = data; + public string File { get; } = file; + + public InvalidDataException Damaged(string what) => new($"{File} is damaged ({what} points outside the file)."); + + private ReadOnlySpan At(int offset, int length) => + offset >= 0 && (long)offset + length <= Data.Length ? Data.AsSpan(offset, length) : throw Damaged("an offset"); + + public int I32(int offset) => BinaryPrimitives.ReadInt32LittleEndian(At(offset, 4)); + public int I16(int offset) => BinaryPrimitives.ReadInt16LittleEndian(At(offset, 2)); + public int U16(int offset) => BinaryPrimitives.ReadUInt16LittleEndian(At(offset, 2)); + public byte Byte(int offset) => At(offset, 1)[0]; + public float F32(int offset) => BinaryPrimitives.ReadSingleLittleEndian(At(offset, 4)); + public Vector3 Vector(int offset) => new(F32(offset), F32(offset + 4), F32(offset + 8)); + + /// A count at , at most . + public int Count(int offset, int max, string what) + { + var count = I32(offset); + if (count < 0 || count > max) + throw new InvalidDataException($"{File} is damaged (it claims {count} {what})."); + return count; + } + + /// A NUL-terminated name at , at most bytes. + public string String(int offset, int max = 1024) + { + if (offset < 0 || offset >= Data.Length) + throw Damaged("a name"); + var span = Data.AsSpan(offset, Math.Min(max, Data.Length - offset)); + var end = span.IndexOf((byte)0); + return Encoding.UTF8.GetString(end < 0 ? span : span[..end]); + } + } +} + +/// +/// Finds a Source model's textures as Garry's Mod does: "materials/<texture folder>/<texture>.vmt" beside +/// the "models" folder the model is in, then that material's $basetexture as a .vtf. Only .vmt and .vtf files below that +/// "materials" folder are read, never through "..". +/// +internal sealed class SourceMaterials +{ + /// How far above the model its "models" folder is looked for (models/a/b/c/model.mdl is four). + private const int MaxLevels = 10; + + private readonly string? _root; + private readonly IReadOnlyList _folders; + private readonly Dictionary _textures = new(StringComparer.OrdinalIgnoreCase); + + public SourceMaterials(string mdlPath, IReadOnlyList folders) + { + _folders = folders.Count > 0 ? folders : [""]; + // Only the game's or addon's own layout: a "models" folder above the model, with "materials" beside it. + var folder = new DirectoryInfo(Path.GetDirectoryName(Path.GetFullPath(mdlPath))!); + for (var level = 0; level < MaxLevels && folder.Parent is { } parent; level++, folder = parent) + { + if (!folder.Name.Equals("models", StringComparison.OrdinalIgnoreCase)) + continue; + try + { + _root = parent.EnumerateDirectories() + .FirstOrDefault(d => d.Name.Equals("materials", StringComparison.OrdinalIgnoreCase))?.FullName; + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + // a folder Filee may not list: no textures + } + break; + } + } + + /// The texture of material , and whether the material is drawn from both sides. + public (Texture3D? Texture, bool DoubleSided) Find(string name) + { + if (_root is null) + return (null, false); + foreach (var folder in _folders) + { + var relative = folder.Length > 0 ? folder + "/" + name : name; + if (Resolve(relative + ".vmt") is { } vmt) + { + var material = Vmt.Read(vmt, path => Resolve(path.StartsWith("materials/", StringComparison.OrdinalIgnoreCase) ? path[10..] : path)); + var texture = material.BaseTexture is { } baseTexture ? Texture(StripExtension(baseTexture) + ".vtf") : null; + return (texture, material.NoCull); + } + if (Resolve(relative + ".vtf") is not null) + return (Texture(relative + ".vtf"), false); + } + return (null, false); + } + + private static string StripExtension(string path) => + path.EndsWith(".vtf", StringComparison.OrdinalIgnoreCase) ? path[..^4] : path; + + /// The decoded texture at (below "materials"), as PNG, once per file. + private Texture3D? Texture(string relative) + { + if (_textures.TryGetValue(relative, out var known)) + return known; + Texture3D? texture = null; + if (Resolve(relative) is { } path) + { + try + { + using var frames = Vtf.Read(path); + if (frames.Count > 0) + texture = new Texture3D(frames[0].ToByteArray(MagickFormat.Png), "png", Path.GetFileNameWithoutExtension(path)); + } + catch (Exception ex) when (ex is InvalidDataException or MagickException or IOException) + { + // A texture Filee can't read leaves the material plain. + } + } + _textures[relative] = texture; + return texture; + } + + /// + /// The file at below "materials" ("/" or "\" separated), letter case aside (Source's file + /// system ignores it), or null. + /// + private string? Resolve(string relative) + { + var parts = relative.Replace('\\', '/').Split('/', StringSplitOptions.RemoveEmptyEntries); + if (parts.Length == 0 || parts.Any(p => p is "." or "..")) + return null; + var current = _root!; + for (var i = 0; i < parts.Length; i++) + { + var exact = Path.Combine(current, parts[i]); + var last = i == parts.Length - 1; + if (last ? File.Exists(exact) : Directory.Exists(exact)) + { + current = exact; + continue; + } + var entries = last ? Directory.EnumerateFiles(current) : Directory.EnumerateDirectories(current); + var match = entries.FirstOrDefault(e => string.Equals(Path.GetFileName(e), parts[i], StringComparison.OrdinalIgnoreCase)); + if (match is null) + return null; + current = match; + } + return current; + } +} + +/// The few things Filee reads from a Valve material (.vmt, KeyValues text): $basetexture and $nocull. +internal static class Vmt +{ + internal sealed record Material(string? BaseTexture, bool NoCull); + + /// + /// Reads ; a "patch" material takes what it doesn't set from the material it includes + /// (found through , one level). + /// + public static Material Read(string path, Func resolve) + { + var tokens = Tokens(File.ReadAllText(path)); + var material = Parse(tokens); + if (tokens.Count > 0 && tokens[0].Equals("patch", StringComparison.OrdinalIgnoreCase) + && Value(tokens, "include") is { } include && include.EndsWith(".vmt", StringComparison.OrdinalIgnoreCase) + && resolve(include.Replace('\\', '/')) is { } included && included != path) + { + var parent = Parse(Tokens(File.ReadAllText(included))); + material = new Material(material.BaseTexture ?? parent.BaseTexture, material.NoCull || parent.NoCull); + } + return material; + } + + private static Material Parse(List tokens) => + new(Value(tokens, "$basetexture"), Value(tokens, "$nocull") is "1"); + + /// The value after the first key , at any depth. + private static string? Value(List tokens, string key) + { + for (var i = 0; i + 1 < tokens.Count; i++) + { + if (tokens[i].Equals(key, StringComparison.OrdinalIgnoreCase) && tokens[i + 1] is not ("{" or "}")) + return tokens[i + 1]; + } + return null; + } + + /// KeyValues tokens: quoted or bare words and braces; "//" comments are skipped. + internal static List Tokens(string text) + { + var tokens = new List(); + var i = 0; + while (i < text.Length) + { + var c = text[i]; + if (char.IsWhiteSpace(c)) + { + i++; + } + else if (c == '/' && i + 1 < text.Length && text[i + 1] == '/') + { + while (i < text.Length && text[i] != '\n') + i++; + } + else if (c is '{' or '}') + { + tokens.Add(c.ToString()); + i++; + } + else if (c == '"') + { + var end = text.IndexOf('"', i + 1); + if (end < 0) + end = text.Length; + tokens.Add(text[(i + 1)..end]); + i = end + 1; + } + else + { + var start = i; + while (i < text.Length && !char.IsWhiteSpace(text[i]) && text[i] is not ('{' or '}' or '"')) + i++; + tokens.Add(text[start..i]); + } + } + return tokens; + } +} diff --git a/tests/Filee.Core.Tests/FormatFieldsTests.cs b/tests/Filee.Core.Tests/FormatFieldsTests.cs index 0908ad978..bc11aac35 100644 --- a/tests/Filee.Core.Tests/FormatFieldsTests.cs +++ b/tests/Filee.Core.Tests/FormatFieldsTests.cs @@ -26,16 +26,16 @@ private static string[] Hidden(IReadOnlySet turnedOff) => private static readonly string[] HiddenWithEverythingOff = [ "3g2", "3gp", "3mf", "602", "aab", "ac3", "adoc", "ai", "amr", "ani", "ape", "apks", "arj", "ass", "au", "azw", - "azw4", "bik", "bz2", "cab", "caf", "cavs", "cdr", "cgm", "chm", "cpio", "cur", "cwk", "dav", "dbk", "dcm", "dds", "deb", + "azw4", "bik", "bsp", "bz2", "cab", "caf", "cavs", "cdr", "cgm", "chm", "cpio", "cur", "cwk", "dav", "dbk", "dcm", "dds", "deb", "dff", "dot", "dsf", "dss", "dst", "dts", "dv", "dwg", "eac3", "eot", "eps", "exr", "fbx", "fit", "fits", - "flc", "flv", "fodg", "fodp", "fods", "fodt", "fon", "gcode", "geojson", "gerber", "glb", "gltf", "gpx", "gz", "h264", "h265", "har", + "flc", "flv", "fodg", "fodp", "fods", "fodt", "fon", "gcode", "geojson", "gerber", "glb", "gltf", "gma", "gpx", "gz", "h264", "h265", "har", "hdr", "htmlz", "icns", "ipynb", "it", "jar", "jp2", "jsonl", "kml", "kmz", "kra", "lha", "lit", "lrf", "lwp", "lz", - "lzma", "m4s", "man", "mcaddon", "mcpack", "mctemplate", "mcw", "mcworld", "mediawiki", "miff", "mng", "mobi", + "lzma", "m4s", "man", "mcaddon", "mcpack", "mctemplate", "mcw", "mcworld", "mdl", "mediawiki", "miff", "mng", "mobi", "mod", "mp2", "mpc", "mrpack", "mxf", "nov", "nupkg", "obj", "oeb", "org", "osm", "otf", "pam", "pcd", "pcx", "pdb", "pdn", "pfm", "plist", "plt", "ply", "pml", "ppm", "prc", "ps", "psb", "psd", - "psw", "qoi", "ras", "raw", "rb", "rm", "rpm", "rst", "s3m", "scc", "sda", "sdc", "sdw", "sgi", "shp", "snb", "spx", + "psw", "qoi", "ras", "raw", "rb", "rm", "rpm", "rst", "s3m", "scc", "sda", "sdc", "sdw", "sgi", "shp", "smd", "snb", "spx", "ssa", "stl", "sub", "swf", "tak", "tar", "tar7z", "tbz2", "tcr", "tex", "textile", "tgz", "tlz", "toml", - "tta", "ttf", "ttml", "txtz", "txz", "typ", "tz", "usm", "vob", "voc", "vtf", "w64", "wbmp", "wmf", "woff", "woff2", + "tta", "ttf", "ttml", "txtz", "txz", "typ", "tz", "usm", "vob", "voc", "vpk", "vtf", "w64", "wbmp", "wmf", "woff", "woff2", "wpd", "wtv", "wv", "xbm", "xcf", "xlt", "xm", "xpm", "xz", "yaml", "z", "zxp" ]; @@ -45,7 +45,7 @@ private static string[] Hidden(IReadOnlySet turnedOff) => { "model3d", ["fbx", "glb", "gltf", "ply"] }, { "geo", ["fit", "geojson", "gpx", "kml", "kmz", "osm", "shp"] }, { "font", ["otf", "ttf", "woff", "woff2"] }, - { "game", ["bik", "dds", "mcaddon", "mcpack", "mctemplate", "mcworld", "mrpack", "usm", "vtf"] }, + { "game", ["bik", "bsp", "dds", "gma", "mcaddon", "mcpack", "mctemplate", "mcworld", "mdl", "mrpack", "smd", "usm", "vpk", "vtf"] }, { "design", ["ai", "cdr", "dst", "eps", "exr", "hdr", "kra", "pdn", "plt", "ps", "psb", "psd", "raw", "xcf"] }, { "science", ["dcm", "fits"] }, { "developer", [ @@ -72,7 +72,7 @@ private static string[] Hidden(IReadOnlySet turnedOff) => { "model3d", ["3mf", "fbx", "gcode", "glb", "gltf", "obj", "ply", "stl"] }, { "geo", ["fit", "geojson", "gpx", "kml", "kmz", "osm", "shp"] }, { "font", ["eot", "fon", "otf", "ttf", "woff", "woff2"] }, - { "game", ["bik", "dds", "flc", "it", "mcaddon", "mcpack", "mctemplate", "mcworld", "mrpack", "tga", "usm", "vtf", "xm"] }, + { "game", ["bik", "bsp", "dds", "flc", "gma", "it", "mcaddon", "mcpack", "mctemplate", "mcworld", "mdl", "mrpack", "smd", "tga", "usm", "vpk", "vtf", "xm"] }, { "design", ["ai", "cdr", "dst", "eps", "exr", "hdr", "jp2", "kra", "pcd", "pdn", "plt", "ps", "psb", "psd", "raw", "xcf"] }, { "science", ["dcm", "fits", "ipynb", "jp2"] }, { "developer", [ @@ -145,7 +145,7 @@ public void Grade_counts_match_the_published_table() { [FormatGrade.Everyday] = 19, [FormatGrade.General] = 99, - [FormatGrade.Specialist] = 127, + [FormatGrade.Specialist] = 132, [FormatGrade.Legacy] = 52, }, counts); diff --git a/tests/Filee.Core.Tests/SourceFormatTests.cs b/tests/Filee.Core.Tests/SourceFormatTests.cs new file mode 100644 index 000000000..98238ca98 --- /dev/null +++ b/tests/Filee.Core.Tests/SourceFormatTests.cs @@ -0,0 +1,110 @@ +// Source engine files (#1526): Garry's Mod addons, Valve paks and maps (for their packed files) open the archives donut, +// which existing libraries learn without taking a donut of the user's own, and Source models and SMD the 3D models +// donut; each is a target only where the Games field is on. + +using Filee.Core.Formats; +using Filee.Core.Presets; +using Filee.Core.Profiles; +using Filee.Core.Settings; + +namespace Filee.Core.Tests; + +public class SourceFormatTests +{ + [Theory] + [InlineData("wire.gma", "gma", "archives")] + [InlineData("WIRE.GMA", "gma", "archives")] + [InlineData("pak01_dir.vpk", "vpk", "archives")] + [InlineData("gm_construct.bsp", "bsp", "archives")] + [InlineData("gate_e2.mdl", "mdl", "models")] + [InlineData("gate_e2_reference.smd", "smd", "models")] + public void Source_files_use_their_donut(string file, string format, string donut) + { + Assert.Same(FormatRegistry.Get(format), FormatRegistry.Detect(file)); + Assert.Equal(donut, ProfileSelector.Select(BuiltInData.CreateProfiles(), [file])?.Id); + } + + [Theory] + [InlineData("gma", FormatCategory.Archive, true)] + [InlineData("vpk", FormatCategory.Archive, false)] + [InlineData("bsp", FormatCategory.Archive, false)] + [InlineData("mdl", FormatCategory.Model3D, false)] + [InlineData("smd", FormatCategory.Model3D, true)] + public void Source_files_are_game_formats(string id, FormatCategory category, bool writable) + { + var format = FormatRegistry.Get(id)!; + Assert.Equal((category, writable), (format.Category, format.Writable)); + Assert.Equal(FormatGrade.Specialist, FormatFields.GradeOf(format)); + Assert.False(FormatFields.IsShown(id, new HashSet { "game" })); + Assert.True(FormatFields.IsShown(id, new HashSet())); + } + + [Fact] + public void Gma_joins_the_game_packs_and_smd_the_3d_models_in_the_palette() + { + var presets = BuiltInData.CreatePresets(); + var ids = presets.Select(p => p.Id).ToList(); + Assert.Equal(ids.IndexOf("to-mrpack") + 1, ids.IndexOf("to-gma")); + Assert.Equal(ids.IndexOf("to-gltf") + 1, ids.IndexOf("to-smd")); + Assert.Equal("gma", presets.Single(p => p.Id == "to-gma").TargetFormat); + Assert.Equal("smd", presets.Single(p => p.Id == "to-smd").TargetFormat); + } + + [Fact] + public void Version_94_libraries_learn_the_source_files_without_taking_custom_claims() + { + var presets = BuiltInData.CreatePresets(); + var profiles = BuiltInData.CreateProfiles(); + var archives = profiles.Single(p => p.Id == "archives"); + archives.Extensions.RemoveAll(e => e is "gma" or "vpk" or "bsp"); + + Assert.True(SettingsMigrations.ApplyToLibrary(94, presets, profiles)); + Assert.Contains("gma", archives.Extensions); + Assert.Contains("vpk", archives.Extensions); + Assert.Contains("bsp", archives.Extensions); + Assert.Equal(BuiltInData.CreatePresets().Select(p => p.Id), presets.Select(p => p.Id)); // no preset added + Assert.False(SettingsMigrations.ApplyToLibrary(94, presets, profiles)); + + archives.Extensions.Remove("gma"); + profiles.Insert(0, new ToolbarProfile { Id = "my-addons", Extensions = ["GMA"] }); + Assert.False(SettingsMigrations.ApplyToLibrary(94, presets, profiles)); + Assert.DoesNotContain("gma", archives.Extensions); // the user's own donut has it + Assert.Equal("my-addons", ProfileSelector.Select(profiles, ["wire.gma"])?.Id); + } + + [Fact] + public void Version_95_libraries_get_the_gma_preset_after_the_game_packs() + { + var presets = BuiltInData.CreatePresets(); + presets.RemoveAll(p => p.Id == "to-gma"); + var profiles = BuiltInData.CreateProfiles(); + + Assert.True(SettingsMigrations.ApplyToLibrary(95, presets, profiles)); + var ids = presets.Select(p => p.Id).ToList(); + Assert.Equal(ids.IndexOf("to-mrpack") + 1, ids.IndexOf("to-gma")); + Assert.False(SettingsMigrations.ApplyToLibrary(95, presets, profiles)); + // Not on a donut. + Assert.DoesNotContain(BuiltInData.CreateProfiles(), p => p.PresetIds.Contains("to-gma") || p.BackPresetIds?.Contains("to-gma") == true); + } + + [Fact] + public void Version_96_libraries_learn_source_models_and_get_the_smd_preset() + { + var presets = BuiltInData.CreatePresets(); + var profiles = BuiltInData.CreateProfiles(); + var models = profiles.Single(p => p.Id == "models"); + models.Extensions.RemoveAll(e => e is "mdl" or "smd"); + presets.RemoveAll(p => p.Id == "to-smd"); + + Assert.True(SettingsMigrations.ApplyToLibrary(96, presets, profiles)); + Assert.Contains("mdl", models.Extensions); + Assert.Contains("smd", models.Extensions); + Assert.Equal(BuiltInData.CreatePresets().Select(p => p.Id), presets.Select(p => p.Id)); // "SMD" back after glTF + Assert.False(SettingsMigrations.ApplyToLibrary(96, presets, profiles)); + + models.Extensions.Remove("mdl"); + profiles.Insert(0, new ToolbarProfile { Id = "my-models", Extensions = ["MDL"] }); + Assert.False(SettingsMigrations.ApplyToLibrary(96, presets, profiles)); + Assert.DoesNotContain("mdl", models.Extensions); + } +} diff --git a/tests/Filee.Core.Tests/TranslatePresetMigrationTests.cs b/tests/Filee.Core.Tests/TranslatePresetMigrationTests.cs index 5ae8ea156..c11a2e9c9 100644 --- a/tests/Filee.Core.Tests/TranslatePresetMigrationTests.cs +++ b/tests/Filee.Core.Tests/TranslatePresetMigrationTests.cs @@ -127,7 +127,7 @@ public void Translate_goes_after_the_built_in_preset_before_it() var defaults = BuiltInData.CreatePresets(); List Only(params string[] ids) => ids.Select(id => defaults.Single(p => p.Id == id)).ToList(); // Later steps add their own presets; what is checked here is where Translate lands. - static IEnumerable Ids(List presets) => presets.Select(p => p.Id).Where(id => id is not (BuiltInData.MetadataId or BuiltInData.UpscaleId or BuiltInData.RingtoneId or BuiltInData.PdfAId or "to-iso")); + static IEnumerable Ids(List presets) => presets.Select(p => p.Id).Where(id => id is not (BuiltInData.MetadataId or BuiltInData.UpscaleId or BuiltInData.RingtoneId or BuiltInData.PdfAId or "to-iso" or "to-gma" or "to-smd")); List Profiles() { var profiles = BuiltInData.CreateProfiles(); diff --git a/tests/Filee.Core.Tests/UserDataStoreTests.cs b/tests/Filee.Core.Tests/UserDataStoreTests.cs index 887570306..39a51415a 100644 --- a/tests/Filee.Core.Tests/UserDataStoreTests.cs +++ b/tests/Filee.Core.Tests/UserDataStoreTests.cs @@ -1060,7 +1060,7 @@ public void Version_14_libraries_get_3d_model_presets_and_their_donut_after_data Assert.Equal(ids.IndexOf("models") + 1, ids.IndexOf("geo")); // and, from schema 25, the geospatial donut Assert.Equal(ids.IndexOf("mixed") - 1, ids.IndexOf("geo")); var models = profiles.Single(p => p.Id == "models"); - Assert.Equal(["glb", "gltf", "stl", "obj", "3mf", "ply", "fbx"], models.Extensions); + Assert.Equal(["glb", "gltf", "stl", "obj", "3mf", "ply", "fbx", "mdl", "smd"], models.Extensions); Assert.Equal(BuiltInData.ModelProfile().PresetIds, models.PresetIds); Assert.False(SettingsMigrations.ApplyToLibrary(14, presets, profiles)); // once only } @@ -1074,7 +1074,7 @@ public void The_3d_donut_leaves_out_extensions_a_profile_of_the_user_takes() Assert.True(SettingsMigrations.ApplyToLibrary(14, presets, profiles)); - Assert.Equal(["glb", "gltf", "obj", "ply", "fbx"], profiles.Single(p => p.Id == "models").Extensions); + Assert.Equal(["glb", "gltf", "obj", "ply", "fbx", "mdl", "smd"], profiles.Single(p => p.Id == "models").Extensions); } [Fact] @@ -1088,7 +1088,7 @@ public void Version_21_libraries_learn_fbx_on_the_3d_donut_and_nothing_else() Assert.True(SettingsMigrations.ApplyToLibrary(21, presets, profiles)); - Assert.Equal(["glb", "gltf", "stl", "obj", "3mf", "fbx"], models.Extensions); + Assert.Equal(["glb", "gltf", "stl", "obj", "3mf", "mdl", "smd", "fbx"], models.Extensions); Assert.False(SettingsMigrations.ApplyToLibrary(21, presets, profiles)); // once only // A donut of the user that already opens FBX files keeps them. diff --git a/tests/Filee.Engines.Tests/BspTests.cs b/tests/Filee.Engines.Tests/BspTests.cs new file mode 100644 index 000000000..dcc69b5ef --- /dev/null +++ b/tests/Filee.Engines.Tests/BspTests.cs @@ -0,0 +1,143 @@ +// Source engine maps (.bsp, #1526): the files packed into a map are a ZIP in its pakfile lump (lump 40). It unpacks like +// an archive; Left 4 Dead 2's lump order (version, offset, length) is read too. The test maps are built here: a header, +// some lump data and the ZIP. + +using System.Buffers.Binary; +using System.IO.Compression; +using Filee.Core.Conversion; +using Filee.Core.Formats; +using Filee.Core.Presets; +using Filee.Engines.Archives; +using static Filee.Engines.Tests.ArchiveBuilders; + +namespace Filee.Engines.Tests; + +public class BspTests(EngineFixture fx) : IClassFixture +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private void SkipWithout7Zip() + { + var engine = fx.Converters.Single(c => c.Id == "archive"); + Assert.SkipUnless(fx.Catalog.StatusOf(engine).IsAvailable, "7-Zip not found (pwsh build/fetch-engines.ps1 -Only 7zip)"); + } + + private async Task ConvertOkAsync(string source, string presetId) + { + var job = await fx.ConvertAsync([source], BuiltInData.CreatePresets().Single(p => p.Id == presetId)); + Assert.True(job.State == JobState.Completed, string.Join("; ", job.Files.Select(f => $"{f.ErrorKey} {f.ErrorDetail}"))); + return Assert.Single(job.Outputs); + } + + /// A ZIP as VBSP packs it: stored, with the map's custom files. + private static byte[] PakfileZip(params (string Name, byte[] Data)[] files) + { + using var stream = new MemoryStream(); + using (var zip = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true)) + { + foreach (var (name, data) in files) + { + using var entry = zip.CreateEntry(name, CompressionLevel.NoCompression).Open(); + entry.Write(data); + } + } + return stream.ToArray(); + } + + /// + /// A map of version : the header, 4 KB of other lump data, then + /// as lump 40 (in Left 4 Dead 2's order when ). + /// + private static byte[] BuildMap(int version, byte[] pakfile, bool l4d2 = false) + { + const int headerSize = 8 + 64 * 16 + 4; + var other = Noise(4096, 7); + var map = new byte[headerSize + other.Length + pakfile.Length]; + "VBSP"u8.CopyTo(map); + BinaryPrimitives.WriteInt32LittleEndian(map.AsSpan(4), version); + // Lump 0 (entities) points at the other data, as in a real map. + BinaryPrimitives.WriteInt32LittleEndian(map.AsSpan(8), headerSize); + BinaryPrimitives.WriteInt32LittleEndian(map.AsSpan(12), other.Length); + var lump = map.AsSpan(8 + 40 * 16, 16); + var offset = headerSize + other.Length; + if (l4d2) + { + BinaryPrimitives.WriteInt32LittleEndian(lump, 0); + BinaryPrimitives.WriteInt32LittleEndian(lump[4..], offset); + BinaryPrimitives.WriteInt32LittleEndian(lump[8..], pakfile.Length); + } + else + { + BinaryPrimitives.WriteInt32LittleEndian(lump, offset); + BinaryPrimitives.WriteInt32LittleEndian(lump[4..], pakfile.Length); + } + other.CopyTo(map, headerSize); + pakfile.CopyTo(map, offset); + return map; + } + + private static readonly (string Name, byte[] Data)[] Packed = + [ + ("materials/maps/gm_test/brick.vtf", Noise(2000, 1)), + ("materials/maps/gm_test/brick.vmt", "\"LightmappedGeneric\" { \"$basetexture\" \"maps/gm_test/brick\" }"u8.ToArray()), + ("sound/ambient/wind.wav", Noise(800, 2)), + ]; + + [Fact] + public void Bsp_is_recognised_and_unpacks_in_one_step() + { + Assert.Equal("bsp", FormatRegistry.Detect("gm_construct.BSP")?.Id); + var planner = fx.Catalog.CreatePlanner(["archive"]); + Assert.Single(planner.Plan("bsp", FormatRegistry.Folder)!.Steps); + Assert.Single(planner.Plan("bsp", "zip")!.Steps); + } + + [Theory] + [InlineData(20, false)] + [InlineData(21, true)] + public async Task A_maps_packed_files_unpack(int version, bool l4d2) + { + SkipWithout7Zip(); + var path = Path.Combine(fx.NewFolder(), "gm_test.bsp"); + File.WriteAllBytes(path, BuildMap(version, PakfileZip(Packed), l4d2)); + + var folder = await ConvertOkAsync(path, "extract"); + + foreach (var (name, data) in Packed) + Assert.Equal(data, File.ReadAllBytes(Path.Combine(folder, name))); + } + + [Fact] + public async Task A_maps_packed_files_become_a_zip() + { + SkipWithout7Zip(); + var path = Path.Combine(fx.NewFolder(), "gm_test.bsp"); + File.WriteAllBytes(path, BuildMap(20, PakfileZip(Packed))); + + var zip = await ConvertOkAsync(path, "to-zip"); + + using var archive = ZipFile.OpenRead(zip); + Assert.Equal(Packed.Select(p => p.Name).Order(StringComparer.Ordinal), + archive.Entries.Where(e => !e.FullName.EndsWith('/')).Select(e => e.FullName).Order(StringComparer.Ordinal)); + } + + [Fact] + public void A_map_without_packed_files_and_a_quake_map_are_refused_clearly() + { + var folder = fx.NewFolder(); + + var empty = Path.Combine(folder, "empty.bsp"); + File.WriteAllBytes(empty, BuildMap(20, PakfileZip())); + var error = Assert.Throws(() => Bsp.ExtractPakfile(empty, Path.Combine(folder, "a.zip"), Ct)); + Assert.Contains("no files packed", error.Message); + + var quake = Path.Combine(folder, "e1m1.bsp"); + File.WriteAllBytes(quake, [29, 0, 0, 0, .. Noise(2000, 3)]); // Quake's BSP starts with its version + error = Assert.Throws(() => Bsp.ExtractPakfile(quake, Path.Combine(folder, "b.zip"), Ct)); + Assert.Contains("not a Source engine map", error.Message); + + var cut = Path.Combine(folder, "cut.bsp"); + File.WriteAllBytes(cut, BuildMap(20, PakfileZip(Packed))[..5000]); + Assert.Throws(() => Bsp.ExtractPakfile(cut, Path.Combine(folder, "c.zip"), Ct)); + } +} diff --git a/tests/Filee.Engines.Tests/GmaPackTests.cs b/tests/Filee.Engines.Tests/GmaPackTests.cs new file mode 100644 index 000000000..c43bef291 --- /dev/null +++ b/tests/Filee.Engines.Tests/GmaPackTests.cs @@ -0,0 +1,199 @@ +// Packing Garry's Mod addons (.gma, #1526): an addon folder packs into a GMA as gmad does (addon.json for the title, type, +// tags and ignores; only the files Garry's Mod loads; lower-case names; a CRC per file and over the whole addon), and an +// unpacked addon packs back into the same addon. + +using System.Buffers.Binary; +using System.IO.Compression; +using System.Text; +using System.Text.Json.Nodes; +using Filee.Core.Conversion; +using Filee.Core.Presets; +using Filee.Engines.Archives; +using static Filee.Engines.Tests.ArchiveBuilders; + +namespace Filee.Engines.Tests; + +public class GmaPackTests(EngineFixture fx) : IClassFixture +{ + private async Task ConvertOkAsync(IReadOnlyList sources, Preset preset) + { + var job = await fx.ConvertAsync(sources, preset); + Assert.True(job.State == JobState.Completed, string.Join("; ", job.Files.Select(f => $"{f.ErrorKey} {f.ErrorDetail}"))); + return Assert.Single(job.Outputs); + } + + private static Preset Extract => BuiltInData.CreatePresets().Single(p => p.Id == "extract"); + + /// An addon folder: addon.json, a Lua script, a texture, a model part, and files Garry's Mod doesn't load. + private static string AddonFolder(string parent, string name, bool withJson = true) + { + var folder = Directory.CreateDirectory(Path.Combine(parent, name)).FullName; + if (withJson) + { + File.WriteAllText(Path.Combine(folder, "addon.json"), """ + { + "title": "Wire Gates", + "type": "Tool", + "tags": ["Fun", "build"], + "description": "Logic gates.", + "ignore": ["*.psd", "src/*"] + } + """); + } + Write(folder, "lua/autorun/Gates.lua", Encoding.UTF8.GetBytes("print('gates')\n")); + Write(folder, "materials/gates/gate.vtf", Noise(300, 1)); + Write(folder, "materials/gates/gate.vmt", Encoding.UTF8.GetBytes("\"VertexLitGeneric\" { \"$basetexture\" \"gates/gate\" }")); + Write(folder, "models/gates/gate.mdl", Noise(200, 2)); + Write(folder, "models/gates/gate.sw.vtx", Noise(50, 3)); // unused by the game: whitelist takes it back + Write(folder, "materials/gates/gate.psd", Noise(100, 4)); // ignored by addon.json + Write(folder, "readme.txt", Encoding.UTF8.GetBytes("not loaded by the game")); + return folder; + } + + private static void Write(string folder, string name, byte[] data) + { + var path = Path.Combine(folder, name); + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + File.WriteAllBytes(path, data); + } + + private static Gma.Header ReadHeader(string gma) + { + using var stream = File.OpenRead(gma); + return Gma.ReadHeader(stream, Path.GetFileName(gma)); + } + + [Fact] + public async Task An_addon_folder_packs_as_gmad_does() + { + var folder = AddonFolder(fx.NewFolder(), "wire-gates"); + + var gma = await ConvertOkAsync([folder], new Preset { TargetFormat = "gma", Archive = { CombineIntoOne = true } }); + + Assert.Equal("wire-gates.gma", Path.GetFileName(gma)); + var bytes = File.ReadAllBytes(gma); + Assert.Equal("GMAD"u8.ToArray(), bytes[..4]); + Assert.Equal(3, bytes[4]); + // The CRC of everything before the last four bytes. + Assert.Equal(Crc32.Compute(bytes.AsSpan(0, bytes.Length - 4)), BinaryPrimitives.ReadUInt32LittleEndian(bytes.AsSpan(bytes.Length - 4))); + + var header = ReadHeader(gma); + Assert.Equal("Wire Gates", header.Title); + Assert.Equal("Logic gates.", header.Description); + Assert.Equal("tool", header.Type); + Assert.Equal(new[] { "fun", "build" }, header.Tags); + // Lower case, sorted, only what Garry's Mod loads: no addon.json, no .sw.vtx, no ignored .psd, no readme. + Assert.Equal( + ["lua/autorun/gates.lua", "materials/gates/gate.vmt", "materials/gates/gate.vtf", "models/gates/gate.mdl"], + header.Entries.Select(e => e.Name)); + var texture = header.Entries.Single(e => e.Name.EndsWith(".vtf", StringComparison.Ordinal)); + Assert.Equal(300, texture.Size); + Assert.Equal(Crc32.Compute(Noise(300, 1)), texture.Crc); + } + + [Fact] + public async Task A_folder_without_addon_json_takes_its_name_as_the_title() + { + var folder = AddonFolder(fx.NewFolder(), "My Props", withJson: false); + + var header = ReadHeader(await ConvertOkAsync([folder], new Preset { TargetFormat = "gma", Archive = { CombineIntoOne = true } })); + + Assert.Equal("My Props", header.Title); + Assert.Equal("servercontent", header.Type); + // Without addon.json nothing is ignored by it, but the whitelist still applies. + Assert.DoesNotContain(header.Entries, e => e.Name.EndsWith(".psd", StringComparison.Ordinal)); + Assert.Contains(header.Entries, e => e.Name == "models/gates/gate.mdl"); + } + + [Fact] + public async Task A_folder_with_nothing_garrys_mod_loads_is_refused() + { + var folder = Directory.CreateDirectory(Path.Combine(fx.NewFolder(), "notes")).FullName; + File.WriteAllText(Path.Combine(folder, "readme.txt"), "hello"); + + var job = await fx.ConvertAsync([folder], new Preset { TargetFormat = "gma", Archive = { CombineIntoOne = true } }); + + Assert.Equal(JobState.Failed, job.State); + Assert.Empty(job.Outputs); + } + + [Fact] + public async Task An_addon_unpacks_and_packs_back_into_the_same_addon() + { + SkipWithout7Zip(); + var folder = AddonFolder(fx.NewFolder(), "wire-gates"); + var gma = await ConvertOkAsync([folder], new Preset { TargetFormat = "gma", Archive = { CombineIntoOne = true } }); + + var unpacked = await ConvertOkAsync([gma], Extract); + + Assert.Equal(Encoding.UTF8.GetBytes("print('gates')\n"), File.ReadAllBytes(Path.Combine(unpacked, "lua", "autorun", "gates.lua"))); + Assert.Equal(Noise(300, 1), File.ReadAllBytes(Path.Combine(unpacked, "materials", "gates", "gate.vtf"))); + var json = JsonNode.Parse(File.ReadAllText(Path.Combine(unpacked, "addon.json")))!; + Assert.Equal("Wire Gates", (string?)json["title"]); + Assert.Equal("tool", (string?)json["type"]); + Assert.Equal("Logic gates.", (string?)json["description"]); + + var again = ReadHeader(await ConvertOkAsync([unpacked], new Preset { TargetFormat = "gma", Archive = { CombineIntoOne = true } })); + var first = ReadHeader(gma); + Assert.Equal(first.Title, again.Title); + Assert.Equal(first.Type, again.Type); + Assert.Equal(first.Tags, again.Tags); + Assert.Equal(first.Entries.Select(e => (e.Name, e.Size, e.Crc)), again.Entries.Select(e => (e.Name, e.Size, e.Crc))); + } + + private void SkipWithout7Zip() + { + var engine = fx.Converters.Single(c => c.Id == "archive"); + Assert.SkipUnless(fx.Catalog.StatusOf(engine).IsAvailable, "7-Zip not found (pwsh build/fetch-engines.ps1 -Only 7zip)"); + } + + [Fact] + public async Task A_zip_of_an_addon_folder_without_addon_json_packs_from_inside_it() + { + SkipWithout7Zip(); + var parent = fx.NewFolder(); + // As addons are shared: "My Props/lua/…" with a readme beside the folder, no addon.json. + var wrapper = Directory.CreateDirectory(Path.Combine(parent, "download")).FullName; + AddonFolder(wrapper, "My Props", withJson: false); + File.WriteAllText(Path.Combine(wrapper, "README.txt"), "install into garrysmod/addons"); + var zip = Path.Combine(parent, "my-props.zip"); + ZipFile.CreateFromDirectory(wrapper, zip); + + var header = ReadHeader(await ConvertOkAsync([zip], new Preset { TargetFormat = "gma" })); + + Assert.Contains(header.Entries, e => e.Name == "lua/autorun/gates.lua"); + Assert.Contains(header.Entries, e => e.Name == "models/gates/gate.mdl"); + Assert.DoesNotContain(header.Entries, e => e.Name.Contains("props", StringComparison.Ordinal) || e.Name.EndsWith(".txt", StringComparison.Ordinal)); + } + + [Fact] + public void The_addon_root_is_found_through_wrapper_folders() + { + var parent = fx.NewFolder(); + var nested = Directory.CreateDirectory(Path.Combine(parent, "a", "b", "My Addon", "materials", "x")).FullName; + File.WriteAllBytes(Path.Combine(nested, "y.vtf"), Noise(10, 1)); + File.WriteAllText(Path.Combine(parent, "a", "readme.txt"), "hi"); + Assert.Equal(Path.Combine(parent, "a", "b", "My Addon"), Gma.Root(parent)); + + // addon.json marks the root even without a known folder; a folder with several subfolders is the root itself. + var withJson = Directory.CreateDirectory(Path.Combine(fx.NewFolder(), "wrap", "addon")).FullName; + File.WriteAllText(Path.Combine(withJson, "addon.json"), "{}"); + Assert.Equal(withJson, Gma.Root(Path.GetDirectoryName(withJson)!)); + var several = fx.NewFolder(); + Directory.CreateDirectory(Path.Combine(several, "one")); + Directory.CreateDirectory(Path.Combine(several, "two")); + Assert.Equal(several, Gma.Root(several)); + } + + [Theory] + [InlineData("lua/autorun/x.lua", true)] + [InlineData("materials/a/b/c.vtf", true)] + [InlineData("models/a.dx90.vtx", true)] + [InlineData("models/a.sw.vtx", false)] + [InlineData("gamemodes/sandbox/sandbox.txt", true)] + [InlineData("gamemodes/sandbox/sub/x.txt", false)] + [InlineData("data_static/x.json", true)] + [InlineData("readme.txt", false)] + [InlineData("lua/x.luac", false)] + public void The_whitelist_matches_gmads(string name, bool allowed) => Assert.Equal(allowed, Gma.IsAllowed(name)); +} diff --git a/tests/Filee.Engines.Tests/GmaTests.cs b/tests/Filee.Engines.Tests/GmaTests.cs new file mode 100644 index 000000000..ecfda3c21 --- /dev/null +++ b/tests/Filee.Engines.Tests/GmaTests.cs @@ -0,0 +1,172 @@ +// Garry's Mod addons (.gma, #1526): an addon unpacks into its files with every CRC checked (0, from gmad -nocrc, is not +// checked) and an addon.json for its title, description, type and tags, as gmad reads it to pack the folder again. Names +// that would leave the output folder and damaged addons are refused. The test addons are built here byte by byte. + +using System.Buffers.Binary; +using System.IO.Compression; +using System.Text; +using System.Text.Json.Nodes; +using Filee.Core.Conversion; +using Filee.Core.Formats; +using Filee.Core.Presets; +using Filee.Engines.Archives; +using static Filee.Engines.Tests.ArchiveBuilders; + +namespace Filee.Engines.Tests; + +public class GmaTests(EngineFixture fx) : IClassFixture +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private void SkipWithout7Zip() + { + var engine = fx.Converters.Single(c => c.Id == "archive"); + Assert.SkipUnless(fx.Catalog.StatusOf(engine).IsAvailable, "7-Zip not found (pwsh build/fetch-engines.ps1 -Only 7zip)"); + } + + private async Task ConvertOkAsync(string source, string presetId) + { + var job = await fx.ConvertAsync([source], BuiltInData.CreatePresets().Single(p => p.Id == presetId)); + Assert.True(job.State == JobState.Completed, string.Join("; ", job.Files.Select(f => $"{f.ErrorKey} {f.ErrorDetail}"))); + return Assert.Single(job.Outputs); + } + + private static readonly (string Name, byte[] Data)[] Files = + [ + ("lua/autorun/gates.lua", "print('gates')\n"u8.ToArray()), + ("materials/gates/gate.vtf", Noise(300, 1)), + ("models/gates/gate.mdl", Noise(200, 2)), + ]; + + /// + /// A version 3 addon as gmad writes it: one required-content entry, , a CRC per file + /// (0 for each with off, as gmad -nocrc) and a CRC over the whole addon. + /// + private static byte[] BuildGma((string Name, byte[] Data)[] files, string description, bool crcs = true) + { + using var stream = new MemoryStream(); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + writer.Write("GMAD"u8); + writer.Write((byte)3); + writer.Write(0UL); + writer.Write(1_600_000_000UL); + writer.Write("base\0"u8); // one required content entry + writer.Write((byte)0); + writer.Write("Wire Gates\0"u8); + writer.Write(Encoding.UTF8.GetBytes(description + "\0")); + writer.Write("Author Name\0"u8); + writer.Write(1); + var number = 0u; + foreach (var (name, data) in files) + { + writer.Write(++number); + writer.Write(Encoding.UTF8.GetBytes(name + "\0")); + writer.Write((long)data.Length); + writer.Write(crcs ? Crc32.Compute(data) : 0u); + } + writer.Write(0u); + foreach (var (_, data) in files) + writer.Write(data); + } + var bytes = stream.ToArray(); + return [.. bytes, .. BitConverter.GetBytes(crcs ? Crc32.Compute(bytes) : 0u)]; + } + + private const string JsonDescription = """{ "description": "Logic gates.", "type": "tool", "tags": ["fun", "build"] }"""; + + [Fact] + public void Gma_is_recognised_and_packs_and_unpacks_in_one_step() + { + Assert.Equal("gma", FormatRegistry.Detect("Addon.GMA")?.Id); + Assert.True(FormatRegistry.Get("gma")!.Writable); + var planner = fx.Catalog.CreatePlanner(["archive"]); + Assert.Single(planner.Plan("gma", "zip")!.Steps); + Assert.Single(planner.Plan("gma", FormatRegistry.Folder)!.Steps); + Assert.Single(planner.Plan("zip", "gma")!.Steps); + Assert.Single(planner.Plan(FormatRegistry.Folder, "gma")!.Steps); + } + + [Fact] + public async Task An_addon_unpacks_with_an_addon_json_gmad_packs_it_from() + { + SkipWithout7Zip(); + var path = Path.Combine(fx.NewFolder(), "wire-gates.gma"); + File.WriteAllBytes(path, BuildGma(Files, JsonDescription)); + + var folder = await ConvertOkAsync(path, "extract"); + + foreach (var (name, data) in Files) + Assert.Equal(data, File.ReadAllBytes(Path.Combine(folder, name))); + var json = JsonNode.Parse(File.ReadAllText(Path.Combine(folder, "addon.json")))!; + Assert.Equal("Wire Gates", (string?)json["title"]); + Assert.Equal("tool", (string?)json["type"]); + Assert.Equal("Logic gates.", (string?)json["description"]); + Assert.Equal(["fun", "build"], json["tags"]!.AsArray().Select(t => (string?)t)); + } + + [Fact] + public async Task An_addon_without_checksums_becomes_a_zip() + { + SkipWithout7Zip(); + var path = Path.Combine(fx.NewFolder(), "old.gma"); + File.WriteAllBytes(path, BuildGma([("lua/a.lua", "x = 1"u8.ToArray()), ("sound/b.wav", Noise(64, 5))], "Just text.", crcs: false)); + + var zip = await ConvertOkAsync(path, "to-zip"); + + using var archive = ZipFile.OpenRead(zip); + Assert.Equal(["addon.json", "lua/a.lua", "sound/b.wav"], + archive.Entries.Where(e => !e.FullName.EndsWith('/')).Select(e => e.FullName).Order(StringComparer.Ordinal)); + // An old addon's description is plain text, kept as it is. + var json = JsonNode.Parse(new StreamReader(archive.GetEntry("addon.json")!.Open()).ReadToEnd())!; + Assert.Equal("Just text.", (string?)json["description"]); + } + + [Theory] + [InlineData("../evil.lua")] + [InlineData("lua/../../evil.lua")] + [InlineData("/etc/evil.lua")] + [InlineData("C:/Windows/evil.lua")] + public void A_name_leaving_the_output_folder_is_refused(string name) + { + var folder = fx.NewFolder(); + var path = Path.Combine(folder, "evil.gma"); + File.WriteAllBytes(path, BuildGma([("lua/fine.lua", "x = 1"u8.ToArray()), (name, "boom"u8.ToArray())], JsonDescription)); + var target = Directory.CreateDirectory(Path.Combine(folder, "out")).FullName; + + var error = Assert.Throws(() => Gma.Extract(path, target, Ct)); + + Assert.Contains("outside the target folder", error.Message); + Assert.False(File.Exists(Path.Combine(folder, "evil.lua"))); + } + + [Fact] + public void A_damaged_addon_is_refused() + { + var folder = fx.NewFolder(); + var good = BuildGma([("lua/a.lua", "x = 1"u8.ToArray())], JsonDescription); + + // Cut inside the file's content. + var cut = Path.Combine(folder, "cut.gma"); + File.WriteAllBytes(cut, good[..^8]); + Assert.Throws(() => Gma.Extract(cut, Directory.CreateDirectory(Path.Combine(folder, "a")).FullName, Ct)); + + // A wrong CRC for the file: it follows the name and the 8-byte size. + var bad = (byte[])good.Clone(); + var nameEnd = bad.AsSpan().IndexOf("lua/a.lua\0"u8) + "lua/a.lua\0"u8.Length; + BinaryPrimitives.WriteUInt32LittleEndian(bad.AsSpan(nameEnd + 8), 0xDEADBEEF); + var wrong = Path.Combine(folder, "wrong.gma"); + File.WriteAllBytes(wrong, bad); + Assert.Throws(() => Gma.Extract(wrong, Directory.CreateDirectory(Path.Combine(folder, "b")).FullName, Ct)); + + // A file larger than the addon itself. + var huge = (byte[])good.Clone(); + BinaryPrimitives.WriteInt64LittleEndian(huge.AsSpan(nameEnd), long.MaxValue / 2); + var tooLarge = Path.Combine(folder, "huge.gma"); + File.WriteAllBytes(tooLarge, huge); + Assert.Throws(() => Gma.Extract(tooLarge, Directory.CreateDirectory(Path.Combine(folder, "c")).FullName, Ct)); + + File.WriteAllBytes(Path.Combine(folder, "not.gma"), "PK\x03\x04"u8.ToArray()); + Assert.Throws(() => Gma.Extract(Path.Combine(folder, "not.gma"), folder, Ct)); + } +} diff --git a/tests/Filee.Engines.Tests/SmdTests.cs b/tests/Filee.Engines.Tests/SmdTests.cs new file mode 100644 index 000000000..73885ebb8 --- /dev/null +++ b/tests/Filee.Engines.Tests/SmdTests.cs @@ -0,0 +1,149 @@ +// Studiomdl data (.smd, #1526): an MDL becomes a reference SMD with its bones, bind pose and weights (as Crowbar +// decompiles it: counter-clockwise, UVs from the bottom left, inches, Z up); other models become a static prop with one +// root bone; an SMD's triangles become a model again. + +using System.Globalization; +using System.Numerics; +using Filee.Core.Conversion; +using Filee.Core.Presets; +using Filee.Engines.Models3D; +using static Filee.Engines.Tests.SourceModelBuilder; + +namespace Filee.Engines.Tests; + +public class SmdTests(EngineFixture fx) : IClassFixture +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private async Task ConvertOkAsync(string source, string target) + { + var job = await fx.ConvertAsync([source], new Preset { TargetFormat = target }); + Assert.True(job.State == JobState.Completed, string.Join("; ", job.Files.Select(f => $"{f.ErrorKey} {f.ErrorDetail}"))); + return Assert.Single(job.Outputs); + } + + /// The lines of a section ("nodes", "skeleton", "triangles") up to its "end". + private static List Section(string[] lines, string name) + { + var start = Array.IndexOf(lines, name); + Assert.True(start >= 0, $"no {name} section"); + return [.. lines.Skip(start + 1).TakeWhile(l => l != "end")]; + } + + private static float[] Numbers(string line) => + [.. line.Split(' ', StringSplitOptions.RemoveEmptyEntries).Select(f => float.Parse(f, CultureInfo.InvariantCulture))]; + + [Fact] + public void Smd_converts_to_and_from_the_3d_formats_in_one_step() + { + var planner = fx.Catalog.CreatePlanner(["model3d"]); + foreach (var (from, to) in new[] { ("mdl", "smd"), ("obj", "smd"), ("glb", "smd"), ("fbx", "smd"), ("smd", "glb"), ("smd", "stl"), ("smd", "obj") }) + Assert.Single(planner.Plan(from, to)!.Steps); + } + + [Fact] + public async Task A_model_becomes_a_reference_smd_with_its_bones_and_weights() + { + Bone[] bones = + [ + new("root", -1, Vector3.Zero, Vector3.Zero), + new("lid", 0, new Vector3(0, 0, 5), new Vector3(0, 0, MathF.PI / 2)), + ]; + var mdl = Write(fx.NewFolder(), "box", "", ["brick", "glass"], TileAndTriangle(weighted: true), bones); + + var smd = await ConvertOkAsync(mdl, "smd"); + + var lines = File.ReadAllLines(smd); + Assert.Equal("version 1", lines[0]); + Assert.Equal(["0 \"root\" -1", "1 \"lid\" 0"], Section(lines, "nodes")); + var skeleton = Section(lines, "skeleton"); + Assert.Equal("time 0", skeleton[0]); + Assert.Equal([1, 0, 0, 5, 0, 0, MathF.PI / 2], Numbers(skeleton[2]), new FloatComparer()); + + var triangles = Section(lines, "triangles"); + Assert.Equal(3 * 4, triangles.Count); // three triangles, each a material line and three vertices + Assert.Equal(["brick", "brick", "glass"], new[] { triangles[0], triangles[4], triangles[8] }); + // The tile's first triangle (0, 1, 2) in Source's order comes reversed: (10, 10, 0), (0, 10, 0), (0, 0, 0). + Assert.Equal([0, 10, 10, 0, 0, 0, 1, 1, 1, 1, 0, 1], Numbers(triangles[1]), new FloatComparer()); + Assert.Equal([0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1], Numbers(triangles[3]), new FloatComparer()); + // Its corner at (10, 0, 0) is weighted 0.75 / 0.25 to both bones; UV (1, 1) from the top is (1, 0) from the bottom. + Assert.Equal([0, 10, 0, 0, 0, 0, 1, 1, 0, 2, 0, 0.75f, 1, 0.25f], Numbers(triangles[5]), new FloatComparer()); + // The triangle above it hangs on the lid. + Assert.Equal(1, Numbers(triangles[9])[0]); + Assert.Equal([1, 1, 1], Numbers(triangles[9])[^3..], new FloatComparer()); + } + + [Fact] + public async Task An_smd_becomes_a_model_facing_the_same_way_as_its_mdl() + { + var mdl = Write(fx.NewFolder(), "box", "", ["brick", "glass"], TileAndTriangle()); + var fromMdl = ModelConverter.Read(mdl, "mdl", null, Ct); + + var smd = await ConvertOkAsync(mdl, "smd"); + var fromSmd = ModelConverter.Read(smd, "smd", null, Ct); + + Assert.Equal(fromMdl.TriangleCount, fromSmd.TriangleCount); + Assert.Equal(fromMdl.Bounds(), fromSmd.Bounds()); + Assert.Equal(["brick", "glass"], fromSmd.Materials.Select(m => m.Name)); + foreach (var part in fromSmd.Parts) + { + for (var t = 0; t < part.TriangleCount; t++) + Assert.True(Vector3.Dot(part.FaceNormal(t), Vector3.UnitY) > 0.99f); + } + + var glb = await ConvertOkAsync(smd, "glb"); + Assert.Equal(3, ModelConverter.Read(glb, "glb", null, Ct).TriangleCount); + } + + [Fact] + public void Another_model_becomes_a_static_prop_and_reads_back() + { + var model = new Model3D(); + model.Materials.Add(new Material3D { Name = "Painted Metal" }); + var part = new MeshPart { Name = "plate", Material = 0, TexCoords = [] }; + part.AddVertex(new Vector3(0, 0, 0), uv: new Vector2(0, 0)); + part.AddVertex(new Vector3(254, 0, 0), uv: new Vector2(1, 0)); + part.AddVertex(new Vector3(0, 0, -254), uv: new Vector2(0, 1)); + part.AddTriangle(0, 1, 2); + model.Parts.Add(part); + + var path = Path.Combine(fx.NewFolder(), "plate.smd"); + using (var stream = File.Create(path)) + SmdFormat.Write(model, stream, Ct); + + var lines = File.ReadAllLines(path); + Assert.Equal(["0 \"root\" -1"], Section(lines, "nodes")); + var triangles = Section(lines, "triangles"); + Assert.Equal("Painted_Metal", triangles[0]); + // 254 mm along X is 10 inches; −254 mm along Z (Y up) is +10 inches along Y (Z up). + Assert.Equal([0, 10, 0, 0], Numbers(triangles[2])[..4], new FloatComparer()); + Assert.Equal([0, 0, 10, 0], Numbers(triangles[3])[..4], new FloatComparer()); + Assert.EndsWith(" 1 0 1.000000", triangles[1]); + + var back = SmdFormat.Read(path, null, Ct); + back.Validate(); + Assert.Equal(1, back.TriangleCount); + Assert.True(Vector3.Distance(model.Bounds().Max, back.Bounds().Max) < 0.01f); + Assert.True(Vector3.Distance(model.Bounds().Min, back.Bounds().Min) < 0.01f); + Assert.Equal(new Vector2(1, 0), back.Parts[0].TexCoord(1)); + } + + [Fact] + public void A_broken_smd_is_refused_with_its_line() + { + var path = Path.Combine(fx.NewFolder(), "broken.smd"); + File.WriteAllText(path, "version 1\ntriangles\nmat\n0 1 2 3 0 0 1 0 0\n0 1 2 x 0 0 1 0 0\n"); + var error = Assert.Throws(() => SmdFormat.Read(path, null, Ct)); + Assert.Contains("Line 5", error.Message); + + File.WriteAllText(path, "version 1\nnodes\n0 \"root\" -1\nend\nskeleton\ntime 0\n0 0 0 0 0 0 0\nend\n"); + var animation = Assert.Throws(() => ModelConverter.Read(path, "smd", null, Ct)); + Assert.Contains("no triangles", animation.Message); + } + + private sealed class FloatComparer : IEqualityComparer + { + public bool Equals(float a, float b) => MathF.Abs(a - b) < 0.0005f; + public int GetHashCode(float value) => 0; + } +} diff --git a/tests/Filee.Engines.Tests/SourceModelBuilder.cs b/tests/Filee.Engines.Tests/SourceModelBuilder.cs new file mode 100644 index 000000000..943fce31b --- /dev/null +++ b/tests/Filee.Engines.Tests/SourceModelBuilder.cs @@ -0,0 +1,261 @@ +// Builds Source engine models (.mdl version 48 with its .vvd and .dx90.vtx) byte by byte for the tests: bones, textures +// with one texture folder, one body part with one model, and one mesh per material, each a triangle list in one strip +// group. Layouts as in Valve's studio.h and optimize.h (see Models3D/SourceModel.cs). + +using System.Buffers.Binary; +using System.Numerics; +using System.Text; + +namespace Filee.Engines.Tests; + +internal static class SourceModelBuilder +{ + /// A vertex in Source's axes and units (inches, Z up), UV from the top left, weighted to up to two bones. + public sealed record Vertex(Vector3 Position, Vector3 Normal, Vector2 Uv, int Bone = 0, int Bone2 = -1, float Weight2 = 0); + + /// A mesh: its material (an index into the textures), vertices and triangles in Source's (clockwise) order. + public sealed record Mesh(int Material, Vertex[] Vertices, int[] Triangles); + + /// A bone in its bind pose relative to its parent. + public sealed record Bone(string Name, int Parent, Vector3 Position, Vector3 Rotation); + + public const int Checksum = 0x1EE7; + + /// + /// Writes ".mdl", ".vvd" and ".dx90.vtx" into and returns the .mdl; + /// sets a different checksum in the .vvd; stores LOD 0 as + /// two runs of vertices. + /// + public static string Write(string folder, string name, string textureFolder, string[] textures, Mesh[] meshes, + Bone[]? bones = null, int vvdChecksum = Checksum, bool fixups = false) + { + bones ??= [new Bone("static_prop", -1, Vector3.Zero, Vector3.Zero)]; + Directory.CreateDirectory(folder); + var path = Path.Combine(folder, name + ".mdl"); + File.WriteAllBytes(path, Mdl(name, textureFolder, textures, meshes, bones)); + File.WriteAllBytes(Path.Combine(folder, name + ".vvd"), Vvd(meshes, vvdChecksum, fixups)); + File.WriteAllBytes(Path.Combine(folder, name + ".dx90.vtx"), Vtx(meshes)); + return path; + } + + private static byte[] Mdl(string name, string textureFolder, string[] textures, Mesh[] meshes, Bone[] bones) + { + const int header = 408, bone = 216, texture = 64, bodyPart = 16, model = 148, mesh = 116; + var boneAt = header; + var textureAt = boneAt + bones.Length * bone; + var folderTableAt = textureAt + textures.Length * texture; + var skinAt = folderTableAt + 4; + var bodyPartAt = skinAt + textures.Length * 2 + 2; + var modelAt = bodyPartAt + bodyPart; + var meshAt = modelAt + model; + var stringsAt = meshAt + meshes.Length * mesh; + + var strings = new MemoryStream(); + int Str(string text) + { + var at = stringsAt + (int)strings.Length; + strings.Write(Encoding.ASCII.GetBytes(text + "\0")); + return at; + } + + var data = new byte[stringsAt]; + var span = data.AsSpan(); + "IDST"u8.CopyTo(span); + I32(span, 4, 48); + I32(span, 8, Checksum); + Encoding.ASCII.GetBytes($"test/{name}.mdl").CopyTo(span[12..]); + I32(span, 156, bones.Length); + I32(span, 160, boneAt); + I32(span, 204, textures.Length); + I32(span, 208, textureAt); + I32(span, 212, 1); + I32(span, 216, folderTableAt); + I32(span, 220, textures.Length); + I32(span, 224, 1); + I32(span, 228, skinAt); + I32(span, 232, 1); + I32(span, 236, bodyPartAt); + + for (var i = 0; i < bones.Length; i++) + { + var at = boneAt + i * bone; + I32(span, at, Str(bones[i].Name) - at); + I32(span, at + 4, bones[i].Parent); + Vec(span, at + 32, bones[i].Position); + F32(span, at + 56, 1); // quaternion w + Vec(span, at + 60, bones[i].Rotation); + } + for (var i = 0; i < textures.Length; i++) + { + var at = textureAt + i * texture; + I32(span, at, Str(textures[i]) - at); + BinaryPrimitives.WriteInt16LittleEndian(span[(skinAt + i * 2)..], (short)i); + } + I32(span, folderTableAt, Str(textureFolder)); + I32(span, bodyPartAt, Str("body") - bodyPartAt); + I32(span, bodyPartAt + 4, 1); + I32(span, bodyPartAt + 8, 1); + I32(span, bodyPartAt + 12, modelAt - bodyPartAt); + Encoding.ASCII.GetBytes(name).CopyTo(span[modelAt..]); + I32(span, modelAt + 72, meshes.Length); + I32(span, modelAt + 76, meshAt - modelAt); + I32(span, modelAt + 80, meshes.Sum(m => m.Vertices.Length)); + I32(span, modelAt + 84, 0); + var first = 0; + for (var i = 0; i < meshes.Length; i++) + { + var at = meshAt + i * mesh; + I32(span, at, meshes[i].Material); + I32(span, at + 4, modelAt - at); + I32(span, at + 8, meshes[i].Vertices.Length); + I32(span, at + 12, first); + first += meshes[i].Vertices.Length; + } + var result = data.Concat(strings.ToArray()).ToArray(); + I32(result, 76, result.Length); + return result; + } + + private static byte[] Vvd(Mesh[] meshes, int checksum, bool fixups) + { + var vertices = meshes.SelectMany(m => m.Vertices).ToArray(); + var fixupCount = fixups ? 2 : 0; + var fixupAt = 64; + var vertexAt = fixupAt + fixupCount * 12; + var data = new byte[vertexAt + vertices.Length * 48]; + var span = data.AsSpan(); + "IDSV"u8.CopyTo(span); + I32(span, 4, 4); + I32(span, 8, checksum); + I32(span, 12, 1); + I32(span, 16, vertices.Length); + I32(span, 48, fixupCount); + I32(span, 52, fixupAt); + I32(span, 56, vertexAt); + I32(span, 60, data.Length); + if (fixups) + { + var half = vertices.Length / 2; + I32(span, fixupAt, 0); + I32(span, fixupAt + 4, 0); + I32(span, fixupAt + 8, half); + I32(span, fixupAt + 12, 0); + I32(span, fixupAt + 16, half); + I32(span, fixupAt + 20, vertices.Length - half); + } + for (var i = 0; i < vertices.Length; i++) + { + var at = vertexAt + i * 48; + var v = vertices[i]; + var two = v.Bone2 >= 0; + F32(span, at, two ? 1 - v.Weight2 : 1); + F32(span, at + 4, two ? v.Weight2 : 0); + span[at + 12] = (byte)v.Bone; + span[at + 13] = (byte)Math.Max(0, v.Bone2); + span[at + 15] = (byte)(two ? 2 : 1); + Vec(span, at + 16, v.Position); + Vec(span, at + 28, v.Normal); + F32(span, at + 40, v.Uv.X); + F32(span, at + 44, v.Uv.Y); + } + return data; + } + + private static byte[] Vtx(Mesh[] meshes) + { + const int header = 36, bodyPart = 8, model = 8, lod = 12, mesh = 9, group = 25, strip = 27; + var bodyPartAt = header; + var modelAt = bodyPartAt + bodyPart; + var lodAt = modelAt + model; + var meshAt = lodAt + lod; + var groupAt = meshAt + meshes.Length * mesh; + var dataAt = groupAt + meshes.Length * group; + + var tail = new MemoryStream(); + var bytes = new byte[dataAt]; + var span = bytes.AsSpan(); + I32(span, 0, 7); + I32(span, 4, 24); + I32(span, 16, Checksum); + I32(span, 20, 1); + I32(span, 28, 1); + I32(span, 32, bodyPartAt); + I32(span, bodyPartAt, 1); + I32(span, bodyPartAt + 4, modelAt - bodyPartAt); + I32(span, modelAt, 1); + I32(span, modelAt + 4, lodAt - modelAt); + I32(span, lodAt, meshes.Length); + I32(span, lodAt + 4, meshAt - lodAt); + for (var i = 0; i < meshes.Length; i++) + { + var m = meshes[i]; + var meshPos = meshAt + i * mesh; + var groupPos = groupAt + i * group; + I32(span, meshPos, 1); + I32(span, meshPos + 4, groupPos - meshPos); + + var vertexPos = dataAt + (int)tail.Length; + foreach (var index in Enumerable.Range(0, m.Vertices.Length)) + { + var vertex = new byte[9]; + vertex[3] = 1; + BinaryPrimitives.WriteUInt16LittleEndian(vertex.AsSpan(4), (ushort)index); + tail.Write(vertex); + } + var indexPos = dataAt + (int)tail.Length; + var u16 = new byte[2]; + foreach (var index in m.Triangles) + { + BinaryPrimitives.WriteUInt16LittleEndian(u16, (ushort)index); + tail.Write(u16); + } + var stripPos = dataAt + (int)tail.Length; + var stripBytes = new byte[strip]; + I32(stripBytes, 0, m.Triangles.Length); + I32(stripBytes, 8, m.Vertices.Length); + BinaryPrimitives.WriteInt16LittleEndian(stripBytes.AsSpan(16), 1); + stripBytes[18] = 0x01; // triangle list + tail.Write(stripBytes); + + I32(span, groupPos, m.Vertices.Length); + I32(span, groupPos + 4, vertexPos - groupPos); + I32(span, groupPos + 8, m.Triangles.Length); + I32(span, groupPos + 12, indexPos - groupPos); + I32(span, groupPos + 16, 1); + I32(span, groupPos + 20, stripPos - groupPos); + } + return [.. bytes, .. tail.ToArray()]; + } + + private static void I32(Span span, int at, int value) => BinaryPrimitives.WriteInt32LittleEndian(span[at..], value); + + private static void F32(Span span, int at, float value) => BinaryPrimitives.WriteSingleLittleEndian(span[at..], value); + + private static void Vec(Span span, int at, Vector3 v) + { + F32(span, at, v.X); + F32(span, at + 4, v.Y); + F32(span, at + 8, v.Z); + } + + /// + /// A 10 × 10 inch floor tile facing up (material 0, four vertices, two triangles) and a triangle 5 inches above it + /// (material 1), both clockwise as Source stores them. + /// + public static Mesh[] TileAndTriangle(bool weighted = false) => + [ + new(0, + [ + new(new(0, 0, 0), Vector3.UnitZ, new(0, 1)), + new(new(0, 10, 0), Vector3.UnitZ, new(0, 0)), + new(new(10, 10, 0), Vector3.UnitZ, new(1, 0)), + new(new(10, 0, 0), Vector3.UnitZ, new(1, 1), weighted ? 0 : 0, weighted ? 1 : -1, weighted ? 0.25f : 0), + ], [0, 1, 2, 0, 2, 3]), + new(1, + [ + new(new(0, 0, 5), Vector3.UnitZ, new(0, 0), weighted ? 1 : 0), + new(new(0, 10, 5), Vector3.UnitZ, new(0, 1), weighted ? 1 : 0), + new(new(10, 10, 5), Vector3.UnitZ, new(1, 1), weighted ? 1 : 0), + ], [0, 1, 2]), + ]; +} diff --git a/tests/Filee.Engines.Tests/SourceModelTests.cs b/tests/Filee.Engines.Tests/SourceModelTests.cs new file mode 100644 index 000000000..61c811ca7 --- /dev/null +++ b/tests/Filee.Engines.Tests/SourceModelTests.cs @@ -0,0 +1,159 @@ +// Source engine models (.mdl with .vvd and .dx90.vtx, #1526): read in their bind pose into Filee's 3D model (Y up, +// millimetres, counter-clockwise), with their textures found through the .vmt files in the "materials" folder above them +// and decoded from VTF. The test models are built by SourceModelBuilder. + +using System.Numerics; +using Filee.Core.Conversion; +using Filee.Core.Formats; +using Filee.Core.Presets; +using Filee.Engines.Magick; +using Filee.Engines.Models3D; +using ImageMagick; + +namespace Filee.Engines.Tests; + +public class SourceModelTests(EngineFixture fx) : IClassFixture +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private async Task> ConvertOkAsync(string source, string target) + { + var job = await fx.ConvertAsync([source], new Preset { TargetFormat = target }); + Assert.True(job.State == JobState.Completed, string.Join("; ", job.Files.Select(f => $"{f.ErrorKey} {f.ErrorDetail}"))); + return [.. job.Outputs]; + } + + /// + /// An addon folder as Garry's Mod mounts it: models/props/tile.mdl (two materials, "brick" and "glass"), the brick's + /// material (in a differently cased folder) pointing at a red 16 × 16 VTF, and no material for the glass. + /// + private string Addon(bool fixups = false) + { + var root = fx.NewFolder(); + var mdl = SourceModelBuilder.Write(Path.Combine(root, "models", "props"), "tile", "models\\props\\", ["brick", "glass"], + SourceModelBuilder.TileAndTriangle(), fixups: fixups); + var materials = Directory.CreateDirectory(Path.Combine(root, "materials", "Models", "Props")).FullName; + File.WriteAllText(Path.Combine(materials, "brick.vmt"), """ + // a comment + "VertexLitGeneric" + { + "$basetexture" "models\props\brick_color" + "$nocull" "1" + } + """); + using var red = new MagickImage(MagickColors.Red, 16, 16); + VtfWriter.Write([red], Path.Combine(materials, "brick_color.vtf")); + return mdl; + } + + [Fact] + public void Mdl_is_a_3d_model_that_converts_in_one_step() + { + Assert.Equal("mdl", FormatRegistry.Detect("Tile.MDL")?.Id); + var planner = fx.Catalog.CreatePlanner(["model3d"]); + foreach (var target in new[] { "obj", "glb", "stl", "3mf", "ply", "gltf" }) + Assert.Single(planner.Plan("mdl", target)!.Steps); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void A_model_is_read_in_filees_axes_units_and_winding(bool fixups) + { + var model = ModelConverter.Read(Addon(fixups), "mdl", null, Ct); + + Assert.Equal(2, model.Parts.Count); + Assert.Equal(3, model.TriangleCount); + Assert.Equal(7, model.VertexCount); + // 10 inches is 254 mm; Source's Z up becomes Y up, its Y becomes −Z. + var (min, max) = model.Bounds(); + Assert.Equal(new Vector3(0, 0, -254), min, new Vector3Comparer()); + Assert.Equal(new Vector3(254, 127, 0), max, new Vector3Comparer()); + foreach (var part in model.Parts) + { + // Counter-clockwise: every face points the way its vertex normals (up) do. + for (var t = 0; t < part.TriangleCount; t++) + Assert.True(Vector3.Dot(part.FaceNormal(t), Vector3.UnitY) > 0.99f); + Assert.All(Enumerable.Range(0, part.VertexCount), v => Assert.Equal(Vector3.UnitY, part.Normal(v), new Vector3Comparer())); + } + // UVs from the bottom left: the tile's corner at the origin had v = 1 from the top. + var tile = model.Parts[0]; + var origin = Enumerable.Range(0, tile.VertexCount).Single(v => tile.Position(v) == Vector3.Zero); + Assert.Equal(Vector2.Zero, tile.TexCoord(origin)); + } + + [Fact] + public void Its_textures_are_found_through_the_materials() + { + var model = ModelConverter.Read(Addon(), "mdl", null, Ct); + + var brick = model.Materials.Single(m => m.Name == "brick"); + Assert.True(brick.DoubleSided); + var texture = Assert.IsType(brick.BaseColorTexture); + Assert.Equal("png", texture.Extension); + using var image = new MagickImage(texture.Data); + Assert.Equal((16u, 16u), (image.Width, image.Height)); + using var pixels = image.GetPixels(); + var pixel = pixels.GetPixel(8, 8)!.ToColor()!; + Assert.InRange(pixel.R, 240, 255); + Assert.InRange(pixel.G, 0, 15); + + // The game's own texture (no material in the addon) leaves the material plain. + Assert.Null(model.Materials.Single(m => m.Name == "glass").BaseColorTexture); + } + + [Fact] + public async Task A_model_becomes_an_obj_with_its_texture_and_a_glb() + { + var mdl = Addon(); + + var obj = await ConvertOkAsync(mdl, "obj"); + Assert.Contains(obj, p => p.EndsWith(".obj", StringComparison.Ordinal)); + Assert.Contains(obj, p => p.EndsWith(".mtl", StringComparison.Ordinal)); + Assert.Contains(obj, p => p.EndsWith(".png", StringComparison.Ordinal)); + + var glb = Assert.Single(await ConvertOkAsync(mdl, "glb")); + var back = ModelConverter.Read(glb, "glb", null, Ct); + Assert.Equal(3, back.TriangleCount); + Assert.Contains(back.Materials, m => m.BaseColorTexture is not null); + } + + [Fact] + public void A_model_without_its_vvd_or_with_another_builds_vvd_is_refused_clearly() + { + var folder = fx.NewFolder(); + var mdl = SourceModelBuilder.Write(folder, "box", "", ["brick", "glass"], SourceModelBuilder.TileAndTriangle()); + File.Delete(Path.ChangeExtension(mdl, ".vvd")); + var missing = Assert.Throws(() => ModelConverter.Read(mdl, "mdl", null, Ct)); + Assert.Contains("box.vvd", missing.Message); + + mdl = SourceModelBuilder.Write(folder, "box2", "", ["brick", "glass"], SourceModelBuilder.TileAndTriangle(), vvdChecksum: 42); + var other = Assert.Throws(() => ModelConverter.Read(mdl, "mdl", null, Ct)); + Assert.Contains("checksum", other.Message); + + var notModel = Path.Combine(folder, "quake.mdl"); + File.WriteAllBytes(notModel, "IDPO"u8.ToArray()); + File.WriteAllBytes(Path.Combine(folder, "quake.vvd"), []); + File.WriteAllBytes(Path.Combine(folder, "quake.dx90.vtx"), []); + Assert.Throws(() => ModelConverter.Read(notModel, "mdl", null, Ct)); + } + + [Fact] + public void Material_files_are_read_as_keyvalues() + { + var tokens = Vmt.Tokens(""" + "UnlitGeneric" // shader + { + $basetexture "a/b c" + "$alphatest" 1 + } + """); + Assert.Equal(["UnlitGeneric", "{", "$basetexture", "a/b c", "$alphatest", "1", "}"], tokens); + } + + private sealed class Vector3Comparer : IEqualityComparer + { + public bool Equals(Vector3 a, Vector3 b) => Vector3.Distance(a, b) < 0.01f; + public int GetHashCode(Vector3 v) => 0; + } +} diff --git a/tests/Filee.Engines.Tests/VpkTests.cs b/tests/Filee.Engines.Tests/VpkTests.cs new file mode 100644 index 000000000..962f68a13 --- /dev/null +++ b/tests/Filee.Engines.Tests/VpkTests.cs @@ -0,0 +1,232 @@ +// Valve paks (.vpk, #1526): the directory file lists the files; their data follows the tree in it (archive 0x7FFF) or is +// in "_nnn.vpk" next to it, after a few preload bytes in the tree. The test paks are built here byte by byte; +// a PS Vita .vpk is a ZIP and goes to 7-Zip. + +using System.Buffers.Binary; +using System.IO.Compression; +using System.Text; +using Filee.Core.Conversion; +using Filee.Core.Formats; +using Filee.Core.Presets; +using Filee.Engines.Archives; +using static Filee.Engines.Tests.ArchiveBuilders; + +namespace Filee.Engines.Tests; + +public class VpkTests(EngineFixture fx) : IClassFixture +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private void SkipWithout7Zip() + { + var engine = fx.Converters.Single(c => c.Id == "archive"); + Assert.SkipUnless(fx.Catalog.StatusOf(engine).IsAvailable, "7-Zip not found (pwsh build/fetch-engines.ps1 -Only 7zip)"); + } + + private async Task ConvertOkAsync(string source, string presetId) + { + var job = await fx.ConvertAsync([source], BuiltInData.CreatePresets().Single(p => p.Id == presetId)); + Assert.True(job.State == JobState.Completed, string.Join("; ", job.Files.Select(f => $"{f.ErrorKey} {f.ErrorDetail}"))); + return Assert.Single(job.Outputs); + } + + /// A file in the pak: its folder, name and extension, content, preload length and where its data goes. + private sealed record PakFile(string Folder, string Name, string Extension, byte[] Content, int Preload, int Archive); + + private const int InDirectory = 0x7FFF; + + /// + /// Writes "_dir.vpk" (version ) and its "_000.vpk" part into + /// ; returns the directory file. + /// + private static string BuildPak(string folder, string name, int version, params PakFile[] files) + { + var tree = new MemoryStream(); + var inDirectory = new MemoryStream(); + var part = new MemoryStream(); + void Text(string s) => tree.Write(Encoding.ASCII.GetBytes(s + "\0")); + var fields = new byte[18]; + foreach (var byExtension in files.GroupBy(f => f.Extension)) + { + Text(byExtension.Key); + foreach (var byFolder in byExtension.GroupBy(f => f.Folder)) + { + Text(byFolder.Key); + foreach (var file in byFolder) + { + Text(file.Name); + var data = file.Content[file.Preload..]; + var target = file.Archive == InDirectory ? inDirectory : part; + BinaryPrimitives.WriteUInt32LittleEndian(fields, Crc32.Compute(file.Content)); + BinaryPrimitives.WriteUInt16LittleEndian(fields.AsSpan(4), (ushort)file.Preload); + BinaryPrimitives.WriteUInt16LittleEndian(fields.AsSpan(6), (ushort)file.Archive); + BinaryPrimitives.WriteUInt32LittleEndian(fields.AsSpan(8), (uint)target.Length); + BinaryPrimitives.WriteUInt32LittleEndian(fields.AsSpan(12), (uint)data.Length); + BinaryPrimitives.WriteUInt16LittleEndian(fields.AsSpan(16), 0xFFFF); + tree.Write(fields); + tree.Write(file.Content.AsSpan(0, file.Preload)); + target.Write(data); + } + Text(""); + } + Text(""); + } + Text(""); + + var dir = new MemoryStream(); + var header = new byte[version == 2 ? 28 : 12]; + BinaryPrimitives.WriteUInt32LittleEndian(header, 0x55AA1234); + BinaryPrimitives.WriteUInt32LittleEndian(header.AsSpan(4), (uint)version); + BinaryPrimitives.WriteUInt32LittleEndian(header.AsSpan(8), (uint)tree.Length); + if (version == 2) + BinaryPrimitives.WriteUInt32LittleEndian(header.AsSpan(12), (uint)inDirectory.Length); + dir.Write(header); + tree.WriteTo(dir); + inDirectory.WriteTo(dir); + var path = Path.Combine(folder, name + "_dir.vpk"); + File.WriteAllBytes(path, dir.ToArray()); + if (part.Length > 0) + File.WriteAllBytes(Path.Combine(folder, name + "_000.vpk"), part.ToArray()); + return path; + } + + private static PakFile[] SampleFiles => + [ + new("materials/brick", "brickwall001", "vtf", Noise(5000, 1), 0, 0), + new("materials/brick", "brickwall001", "vmt", Encoding.ASCII.GetBytes("\"LightmappedGeneric\" { \"$basetexture\" \"brick/brickwall001\" }"), 16, InDirectory), + new("scripts", "game_sounds", "txt", Words(700), 0, InDirectory), + new(" ", "readme", " ", Encoding.ASCII.GetBytes("top level, no extension"), 0, 0), + ]; + + [Fact] + public void Vpk_is_recognised_and_unpacks_in_one_step() + { + Assert.Equal("vpk", FormatRegistry.Detect("pak01_dir.VPK")?.Id); + var planner = fx.Catalog.CreatePlanner(["archive"]); + Assert.Single(planner.Plan("vpk", FormatRegistry.Folder)!.Steps); + Assert.Single(planner.Plan("vpk", "zip")!.Steps); + Assert.False(FormatRegistry.Get("vpk")!.Writable); + } + + [Theory] + [InlineData(1)] + [InlineData(2)] + public async Task A_pak_unpacks_with_its_preload_bytes_and_its_parts(int version) + { + SkipWithout7Zip(); + var dir = BuildPak(fx.NewFolder(), "pak01", version, SampleFiles); + + var folder = await ConvertOkAsync(dir, "extract"); + + foreach (var file in SampleFiles) + { + var name = (file.Folder.Trim() is "" ? "" : file.Folder + "/") + file.Name + (file.Extension.Trim() is "" ? "" : "." + file.Extension); + Assert.Equal(file.Content, File.ReadAllBytes(Path.Combine(folder, name))); + } + } + + [Fact] + public async Task A_pak_becomes_a_zip() + { + SkipWithout7Zip(); + var dir = BuildPak(fx.NewFolder(), "hl2_textures", 2, SampleFiles); + + var zip = await ConvertOkAsync(dir, "to-zip"); + + using var archive = ZipFile.OpenRead(zip); + Assert.Equal(["materials/brick/brickwall001.vmt", "materials/brick/brickwall001.vtf", "readme", "scripts/game_sounds.txt"], + archive.Entries.Where(e => !e.FullName.EndsWith('/')).Select(e => e.FullName).Order(StringComparer.Ordinal)); + } + + [Fact] + public void A_missing_part_a_data_part_and_a_bad_checksum_are_refused_clearly() + { + var folder = fx.NewFolder(); + var dir = BuildPak(folder, "pak01", 2, SampleFiles); + var target = Directory.CreateDirectory(Path.Combine(folder, "out")).FullName; + + var part = Path.Combine(folder, "pak01_000.vpk"); + var error = Assert.Throws(() => Vpk.Extract(part, target, Ct)); + Assert.Contains("pak01_dir.vpk", error.Message); + + File.Move(part, part + ".bak"); + var missing = Assert.Throws(() => Vpk.Extract(dir, Directory.CreateDirectory(Path.Combine(folder, "out2")).FullName, Ct)); + Assert.Contains("pak01_000.vpk", missing.Message); + File.Move(part + ".bak", part); + + // Change one byte of the texture's data in the part. + var bytes = File.ReadAllBytes(part); + bytes[100] ^= 0xFF; + File.WriteAllBytes(part, bytes); + Assert.Throws(() => Vpk.Extract(dir, Directory.CreateDirectory(Path.Combine(folder, "out3")).FullName, Ct)); + + File.WriteAllBytes(dir, File.ReadAllBytes(dir)[..20]); + Assert.Throws(() => Vpk.Extract(dir, Directory.CreateDirectory(Path.Combine(folder, "out4")).FullName, Ct)); + } + + [Theory] + [InlineData("../..", "evil")] + [InlineData("/etc", "evil")] + [InlineData("C:/Windows", "evil")] + public void A_name_leaving_the_output_folder_is_refused(string folderName, string name) + { + var folder = fx.NewFolder(); + var dir = BuildPak(folder, "evil", 1, + new PakFile("lua", "fine", "lua", "x = 1"u8.ToArray(), 0, InDirectory), + new PakFile(folderName, name, "lua", "boom"u8.ToArray(), 0, InDirectory)); + var target = Directory.CreateDirectory(Path.Combine(folder, "a", "b", "out")).FullName; + + var error = Assert.Throws(() => Vpk.Extract(dir, target, Ct)); + + Assert.Contains("outside the target folder", error.Message); + Assert.Empty(Directory.EnumerateFileSystemEntries(target)); // nothing written, not even the safe file + } + + [Fact] + public void A_tree_pointing_outside_itself_is_refused() + { + var folder = fx.NewFolder(); + var dir = BuildPak(folder, "pak01", 1, new PakFile("lua", "a", "lua", Words(100), 10, InDirectory)); + var good = File.ReadAllBytes(dir); + var fieldsAt = good.AsSpan().IndexOf("\0a\0"u8) + 3; // after the name "a", behind "lua\0lua\0" + + // Preload bytes running past the end of the tree. + var preload = (byte[])good.Clone(); + BinaryPrimitives.WriteUInt16LittleEndian(preload.AsSpan(fieldsAt + 4), 60000); + File.WriteAllBytes(dir, preload); + Assert.Throws(() => Vpk.Extract(dir, Directory.CreateDirectory(Path.Combine(folder, "a")).FullName, Ct)); + + // A data part beyond "_999". + var index = (byte[])good.Clone(); + BinaryPrimitives.WriteUInt16LittleEndian(index.AsSpan(fieldsAt + 6), 4000); + File.WriteAllBytes(dir, index); + Assert.Throws(() => Vpk.Extract(dir, Directory.CreateDirectory(Path.Combine(folder, "b")).FullName, Ct)); + + // A tree larger than Filee reads. + var tree = (byte[])good.Clone(); + BinaryPrimitives.WriteUInt32LittleEndian(tree.AsSpan(8), uint.MaxValue); + File.WriteAllBytes(dir, tree); + Assert.Throws(() => Vpk.Extract(dir, Directory.CreateDirectory(Path.Combine(folder, "c")).FullName, Ct)); + } + + [Fact] + public async Task A_ps_vita_package_is_a_zip() + { + SkipWithout7Zip(); + var folder = fx.NewFolder(); + var vpk = Path.Combine(folder, "homebrew.vpk"); + // A real package has eboot.bin beside sce_sys, so the folder is not flattened. + using (var zip = ZipFile.Open(vpk, ZipArchiveMode.Create)) + { + using (var entry = zip.CreateEntry("eboot.bin").Open()) + entry.Write(Noise(32, 8)); + using (var entry = zip.CreateEntry("sce_sys/param.sfo").Open()) + entry.Write(Noise(64, 9)); + } + + var unpacked = await ConvertOkAsync(vpk, "extract"); + + Assert.Equal(Noise(32, 8), File.ReadAllBytes(Path.Combine(unpacked, "eboot.bin"))); + Assert.Equal(Noise(64, 9), File.ReadAllBytes(Path.Combine(unpacked, "sce_sys", "param.sfo"))); + } +}