feat(ktx): add initial ktx input/output support.

[walcht]

Description

Fixes #2329. Add initial KTX2 format support (closest similar plugin/format in OIIO is DDS).

This KTX plugin support obviously nullifies the benefits of using KTX in the
first place. That being said, this plugin is still useful so that end users
don't have to convert back and forth between KTX <-> supported format (e.g., PNG).
It is also useful to convert to and from KTX2 format using OIIO.

An example usecase would be Blender and its glTf import/export plugin.
(see KhronosGroup/glTF-Blender-IO#1896).

Ideally, at some point in the future, OIIO may introduce a new API to accomodate
texture formats that are mainly used for fast texture uploads to GPUs.

For details on KTX2 and what this PR supports and what it doesn't (alongside a
description of current limitations), see src/ktx.imageio/README.md (copied parts
of it at the end of this).

Tests:

Added initial testsuite but only for the input (i.e., info_command) part of the plugin.
Ideally, after fixing CI issues, the tests will be adjusted for ktxouput.

Checklist:

• I have read the guidelines on contributions and code review procedures.
• I have read the Policy on AI Coding Assistants
  => Haven't used any AI coding assistant tools in any capacity whatsoever.
• I have updated the documentation if my PR adds features or changes
  behavior.
• I am sure that this PR's changes are tested in the testsuite.
• I have run and passed the testsuite in CI before submitting the
  PR, by pushing the changes to my fork and seeing that the automated CI
  passed there.
• My code follows the prevailing code style of this project and I
  fixed any problems reported by the clang-format CI test.
• If I added or modified a public C++ API call, I have also amended the
  corresponding Python bindings. If altering ImageBufAlgo functions, I also
  exposed the new functionality as oiiotool options.
  => No new API is introduced. This is just a plugin.

The sections below are copied from src/ktx.imageio/README.md. These will be updated
as this PR progresses.

Supported Encoders/Decoders

• Supported/Tested texture kinds:

  • SINGLE_TEXTURE_1D
  • SINGLE_TEXTURE_2D
  • SINGLE_TEXTURE_3D
  • CUBEMAP_TEXTURE
  • ARRAY_TEXTURE_1D
  • ARRAY_TEXTURE_2D
  • ARRAY_TEXTURE_CUBEMAP (problematic due to OIIO's API)
    [ ] ARRAY_TEXTURE_3D (not planned)

• Supported/Tested raw VkFormats (decoder + encoder):

  • VK_FORMAT_R8_UNORM
  • VK_FORMAT_R8G8_[SRGB|UNORM]
  • VK_FORMAT_R8G8B8_[SRGB|UNORM]
  • VK_FORMAT_R8G8B8A8_[SRGB|UNORM]
  • VK_FORMAT_R16G16B16_SFLOAT
  • VK_FORMAT_ASTC_NxM_SRGB_BLOCK (all block dimensions)
  • VK_FORMAT_ASTC_NxM_SFLOAT_BLOCK (all block dimensions)

• Block-compressed formats (decoder + encoder):

  • ASTC
  • BCn (waiting on libktx BCn support PR merge here: Add BCn encoders/decoders with RDO support KhronosGroup/KTX-Software#1167)

• Basis Universal schemes (encoder + decoder):

  • UASTC
  • ETC1S

• Supercompression schemes (decompressor + compressor):

  • ZLIB
  • ZSTD

Dependencies (only one - libktx)

• libktx: for general KTX@ format support (loading of KTX2 files, transcoding
  support, supercompression decompression support, etc.).

  • Commit hash: see OpenImageIO/src/cmake/build_Ktx.cmake
  • License: Many subresources. All permissive except lib/etcdec.cxx's license (ETC is not used in PR).

This PR should only be merged after the release of libktx 5.0.0.

Resources

• KTX2 Specs: https://registry.khronos.org/KTX/specs/2.0/ktxspec.v2.html
• Official Implementation (KTX-Software): https://github.com/KhronosGroup/KTX-Software
• Basis Universal Supercompression Implementation (used by libktx): https://github.com/BinomialLLC/basis_universal
• Comparing-BCn-texture-decoders: https://aras-p.info/blog/2022/06/23/Comparing-BCn-texture-decoders/

[walcht]

Update: I'm working on integrating BCn decoders/encoders directly within libktx so that this PR gets significantly simpler (e.g., from 12_000 lines most of which are from dependencies to circa 2_000).

[walcht]

Update2: significantly simplified the whole PR. Now only dependency is libktx (since I offloaded most work to libktx via a PR that will be probably merged soon).

Can you please not approve workflows unless I explicitly request a workflow run? (to avoid filling the actions log with CIs that I know will certainly fail).

[lgritz]

Can you please not approve workflows unless I explicitly request a workflow run? (to avoid filling the actions log with CIs that I know will certainly fail).

Sure, in this case I can do so, but only because you are a a first-time contributor to this project, so admins have to approve workflow runs on the project's main account. Once you have any PR accepted and merged, the workflows will run automatically.

Here is the general solution that you or others can do, in this situation and after you no longer need admin approval:

Let's say you have pushed to your fork's branch "mypr" and submitted a PR to the project from that branch. (Aside: I strongly recommend working in topic branches, and never submitting a PR from your "main".)

After submitting the PR, you realize you need to iterate (you want to add things, or CI is not passing and you need to make repairs, or you have review comments to address), but you don't want to do a series of many many pushes as you go, each of which will trigger a CI run on the main branch as well as sending email alerts to everybody "watching" this project, since pushing to your branch is technically a PR update.

Instead, push to a different branch!

git push origin mypr:test

This pushes your fork's "mypr" branch to your fork in a "test" branch. This will run the CI on your fork in your account (allowing you to see if CI passes for the changes you've made), but since your "test" branch is not associated with the PR, it will not send the rest of us alerts and will not trigger a CI run on the project's main repo account. Then when you finally have it right and are ready for the rest of us to see it, you can git push origin mypr to update "mypr", which the system will know is the branch you are making the PR from.

[walcht]

Question about older CMake version that is failing some CIs and icpx:

1. Some CIs require a relatively old version of CMake (3.18.3) but libktx (the library through which KTX2 support is achieved) requires CMake >= 3.23. (e.g., CI oldest clang11/C++17 py3.9 exr3.1 ocio2.3 / c++17 py3.9).
   In case I may not be able to lower libktx's minimum CMake version (e.g., request refused or simply not possible/requires too many changes) => I couldn't lower CMake version simply because libktx dependencies also require CMake >= 3.20. How should I proceed with this? => Solved. Used older libktx version (4.3.2) for these CIs.

2. ICPX compiler is not supported by libktx (haven't tried compiling it with it yet but it is definitely not supported as per the main CMakeList.txt of libktx). How should I proceed here? => Solved. I have just followed libuhrd and overridden C/C++ compilers for libktx in ICX CI.

These are the only* remaining CI issues.

*: there is another one that is Intel-MacOS only but there is a PR already to fix it in libktx.

All CI issues are now solved. Will push updates here when sufficient testing is done.

[walcht]

PR is now ready for initial review. There are still some missing tests but at this point I will wait for initial review then complete missing tests as necessary (there are also a lot of introduced "ktx:<attrib>" attributes. I don't know of any other way to configure the high number of parameters of libktx in ktxoutput).

[lgritz]

I like the general gist of this and I would definitely like to get ktx support in as a 3.2 feature.

Took a preliminary pass over this and commented on an array of topics, not intended to be comprehensive, but enough for you to get started on the obvious things.

[lgritz]

These are the old versions of these call signatures. It's true that many of the existing reader/writers still use them, and we will eventually get around to fixing them.

But for NEW readers that come along, I'd like us to use the new functions from the start. These are the versions of read_native_* that take span<std::byte> and so have explicit bounds to the buffer being passed where it writes into.

Then the old ones (that just take the void* and not a span) can also be trivially implemented as calling the span-based ones (though that merely assumes that the pointer points to enough space).

[walcht]

But for NEW readers that come along, I'd like us to use the new functions from the start. These are the versions of read_native_* that take spanstd::byte and so have explicit bounds to the buffer being passed where it writes into.

I wanted to use the safer span-based implementations initially, but (weirdly?) they don't have a z parameter which is needed by KTX to, for instance, retrieve a particular slice from a 3D volume.

I left these comments underneath in the code (I couldn't find anything in the docs about span-based missing z parameter):

https://github.com/walcht/OpenImageIO/blob/c037288de004eb957b8ebd5ec616515187adfc75/src/ktx.imageio/ktxinput.cpp#L48-L51

    // TODO: why there is no `read_native_scanlines` that takes a span<std::byte>
    // but also a `z` slice index (same as unsafe `read_native_scanlines`)?
    // bool read_native_scanlines(int subimage, int miplevel, int ybegin, int yend,
    //                            span<std::byte> data) override;

[lgritz]

Same comment as on the input side -- since this is an entirely new writer, let's start from day one with the "span-based" versions of these calls fully implemented so we don't need to come back to it later.

[walcht]

Thank you for the initial review. CIs are failing because I was tracking main from KTX-Software. A couple of days ago, libktx 5.0.0-rc2 was finally released which fixes tons of very relevant issues and we can finally stabilize on it.

Some noteworthy notes before any further reviews:

• libktx supports a lot of color spaces (transfer functions and primaries) but currently only sRGB (rec709) and linear color spaces are tested/supported.
• I will be addressing all review comments, Cubemaps support, HDR support, etc. during the next few days (in patches).
• There are tons of formats that libktx supports (e.g., look at VkFormat enum). I will just be focusing on the very few that are very widely used. The others, I will gradually add them in separate PRs.