On-demand part textures

[gotmachine]

This is exploratory work for implementing an on-demand loading mechanism for part textures.

The general idea would be identify which textures are used in parts prior to loading them, then to skip loading them during initial game load, to finally load them selectively and asynchronously when a part is effectively instantiated in game. Textures would then be unloaded when no instantiated part is using them anymore.

The potential benefits would be :

• Reduced initial load time
• Reduced VRAM consumption

However, the caveats are numerous :

• This require a lot of additional loading-time parsing that will at least partially offset the gains. The net result will still likely be positive, but I would temper expectations about this vastly reducing loading times. Even in a worst case scenario, texture loading hardly excess 50 % of the overall loading time (I'd say 30-40% is more typical), and any non-part texture would still be loaded as usual.
• While this can potentially save a lot of VRAM in heavily modded installs having a lot of additional parts, the savings will always be limited to the memory footprint of parts that aren't in use in the current scene.
• This is trading a single upfront time cost for additional work every time a scene is switched, and while the goal is to load textures asynchronously, there will be additional overhead that might end up delaying at which point the game starts to be useable and responsive after a scene switch. I'd say that scene switch delays is one of the worst problem of KSP, so depending on how bad this ends up, I'm not so sure the whole thing would be perceived as an improvement.
• The editor part picker thumbnails uses a rendered copy of the parts models, this wouldn't be possible anymore. A complete reimplementation would be needed. Two options are possible : either keep using rendered models but generate low res versions of the part textures, or use static images. Either way, this mean auto-generating and loading additional textures. The latter is likely the easiest to implement, and KSP already generate static thumbnails anyway for the inventory / cargo part system, so reusing / refactoring that system is likely the best option. The other option (keeping the 3D models but applying a low-res textures) might have been a simpler and better option if those thumbnails didn't exist, but since they do, this would be very wasteful as the scaled down textures will end up eating memory in addition to the memory used by the stock thumbnails[*]. No matter which solution is used, I would estimate this to be a very significant amount of work.
• This will be breaking a major assumption about how things work in stock and consequently is very likely to cause issues in the modding ecosystem. From preliminary research, I think the fallout would be relatively limited, but this is hard to assess. In particular, plugins doing some part rendering without actually instantiating parts are pretty certain to break, I can notably think of DistantObjectEnhancement, maybe KronalVesselViewer too.
• For this to be effective, the implementation needs to be able to know and keep track of every usage of every texture by every part. While this isn't too hard in stock, many mods are implementing various dynamic texture switching mechanisms, and things gets even more hairy with mods implementing non-stock shaders or custom rendering paths (for example, TextureUnlimited). In practice, supporting B9PartSwitch (the mainstream model/texture switchers) is likely doable, the rest will likely stay out of scope. To be clear, "not supporting" such mods would result in the textures they use being loaded as usual instead of on-demand, although there is a potential "missing texture" problem if such a plugin is reusing a texture not used (in a stock way) in the current part but otherwise used (in a stock way) by another part. This might for example be an issue for the "modular stock tanks" plugin the RO gang has put together.
• This being said, it should be possible to provide an API for this feature, allowing other plugins to leverage it. An assessment of potential usages would need to be made, because I would anticipate some complexities and compromises between the shape and usability of that API and our internal bookkeeping requirements. This should likely be assessed after a first minimally viable prototype is put together.

On a side note, I'm not sure I will have the time nor motivation to get this idea to completion. But at least whatever I do will be a base for anyone wanting to pursue this project.

[*] A stretch goal could be to replace the stock cargo/inventory static thumbnails by 3D models with downscaled textures. The stock static thumbnail system generate a 256x256 texture for every part and every variant of every part, which ends up being a significant waste of VRAM. At the very least, the resolution of those thumbs should be reduced, 128x128 would be more than enough given that at 100% UI scale, the viewport for a part is 64x64.

[gotmachine]

Current prototype implementation :

• On loading, first immediate step is to parse all configs and models to build a dictionary of which part is referencing which texture. This include :
  • Parse the part config to find the model(s) used, either implicitly or in the MODEL {} node
  • Also check for eventual texture replacements in the MODEL {} node
  • Then parse all referenced model files (*.mu) to find which textures are defined.
• Then we leverage the fact that KSPCF already re-implement the texture loader to skip loading all found textures.
  • The trick here is to still create the stock GameDatabase.TextureInfo for every texture, but as a derived OnDemandTextureInfo class that will ultimately will be responsible for keeping track of loaded state of every texture. We assign a dummy transparent 1x1 texture to the texture reference, which will in turn be assigned to by whatever code is requesting that texture.
• Then we hook into part compilation to build a final dictionary of every texture used by every AvailablePart. This is done by searching through the textures referenced by the sharedMaterial of every Renderer present on the prefab. We keep a reference to every Material and to every OnDemandTextureInfo it uses.
• When a part is instantiated in-game, by hooking into Part.Awake(), we trigger the load of every texture and swap the reference on the cached Material instances.

This (very incomplete) prototype has shed some light on a few problems :

• Relying on the prefab sharedMaterial would be very straightforward but isn't viable. While the material is effectively shared between the prefab and just instantiated parts, any module can latter decide the part needs its own non-shared Material instance (by simply using the Renderer.material property). Typically, the stock ModulePartVariants will do this on any part having the module, but we can expect this behavior in many other cases, including cases where the .material property has been used on the prefab, in which case the instantiated copies won't share it by default (at least, I think, this need to be verified). This is also problematic if we want a non-blocking async loading mechanism, as it becomes possible for modules to make a copy of the sharedMaterial (by calling .material) before the textures have been swapped.
• This mean there is no way around walking through every renderer, getting every material and every texture reference on the instantiated part, and this will definitely be a rather slow operation, compounding to a significant hiccup in large part count situations, so this likely need to be distributed over multiple frames, with some "allow to steal X ms per frame" mechanism.
• Further prototyping is need to assess how viable this whole idea actually is. Currently, this is all very theoretical and I wouldn't be surprised if some roadblock pops up. The main difficulty here lies with the idea of making texture loading a non-blocking async mechanism where the game is useable and responsive while textures are being loaded. If that idea ends up being not possible, I fear this would put a serious blow to the viability of the whole endeavor. IMO, scene switch delays are much more annoying than the upfront loading time, and making those delays significantly longer to save some VRAM and one minute or two of upfront loading time doesn't feel like a very appealing deal to me, but I guess there is still a population of players with very heavily modded installs that would appreciate it.

Some general remarks, in no particular order :

• We should exclude any non-DDS texture from being made on-demand. Loading and conversion of PNG textures is orders of magnitude slower and is just not viable. We can however leverage the existing KSPCF PNG-to-DDS cache to alleviate that limitation.
• Ideally, we could have a mechanism where disk reads of the texture files to RAM held byte array to be latter uploaded to VRAM are performed asynchronously in a separate thread, like we do in the FastLoader implementation. This would likely significantly reduce how much time untextured parts are visually shown, and would improve responsiveness by lifting some work out of the main thread, improving framerate during that phase, but this obviously adds a lot of complexity. I would qualify this as nice to have, but not necessary in an initial version.
• A minimal viable implementation would need texture unloading as well, in order to make the VRAM savings a thing. The simple way to do that would be by doing some per-texture bookeeping of part instances using them, removing that ref when the part is destroyed, then unloading the texture when no part is referencing it anymore. However, in many situations (typically, quick-loading / scene switching over the same vessel / ship), this would cause a lot of textures to be unloaded and re-loaded immediately. A simple mechanism to avoid that would be to allow textures to stay in VRAM as long as some memory size threshold isn't reached. We can maybe be a bit smart about this, checking actual VRAM contention to determine that threshold, as well as having some user-defined override.
• The current prototype totally ignore textures defined by the stock ModulePartVariant, they will always be loaded as usual. To include them, this require additional upfront parsing of the module config to determine which textures are used. I haven't looked in depth at the matter, but likely, this will also require some special handling to swap textures references in the material references kept by the module after part instantiation. In an ideal world, we would only load textures actually used by the current variants, but this would require additional hooking to perform on-demand loading when the variants are switched. I would put this into the stretch goals category. In any case, equivalent work would have to repeated for any other texture switcher we want to support, namely B9PS.
• Even though swapping textures on prefab referenced materials in an async way isn't a viable option due to non-shared materials usage on part instances, it would still be beneficial to do so when that part has already been instantiated once, as this would prevent having to walk through every material on further instantiations of the same part.

[Phantomical]

So as part of my work with KSPTextureLoader I have become rather familiar with building asset bundles at runtime and I think that ability gives us a much better way to go about this. If we only move two things to be loaded from asset bundles we can actually directly turn on unity's existing mipmap streaming feature, which gives us exactly what we want without needing to figure out which renderers refer to which textures or when we need to actually load/unload them.

To make this work we need 3 parts:

• Textures need to be loaded from asset bundles with mipmap streaming enabled (see Optimize and rewrite texture loading #369)
• Mesh data needs to be loaded from asset bundles so that we can appropriately calculate the UV distribution metrics. Unity 2020.1 and newer have Mesh.RecalculateUVDistributionMetrics but unfortunately the only way for us to get access to it is to build an asset bundle with uv metrics set within.
• We then enable mipmap streaming controllers on all cameras, disable streaming on all textures not used by a part, set the memory budget, and things should mostly "just" work (maybe).

A middle ground might be to enable streaming on textures and then manually track which parts are active and set the requested mipmap level ourselves. We'd need to track textures by part and I expect this to be somewhat difficult to make work well.