Support file-based serialize/deserialize for DiskANN indices using Save/Load traits

[suhasjs]

• Does this PR have a descriptive title that could go in our release notes?
• Does this PR add any new dependencies?
• Does this PR modify any existing APIs?
• Is the change to the API backwards compatible?
• Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

Initial attempt at #737

What does this implement/fix? Briefly explain your changes.

Introduces two new traits (diskann_record::save::Save and diskann_record::load::Load)and a new create diskann-record to support file-based serialization + deserialization. This PR also implements the traits for DiskANNIndex (and all nested structs) for a successful prototype.
Sample output can be found here.

Any other comments?

This PR is meant to be in a draft state until we're happy with the abstractions and the results.

[hildebrandmw]

Thanks Suhas! My comments this round are mainly about enum handling. I think we can do this more cleanly by having enums "handle themselves" rather than lifting it into a somewhat fragile interaction in the Save/Load traits.

Other than that, the code inside diskann-record could use considerable hardening and testing.

Do we want to plan out how plugins like VFS would work in this PR, or save that for the future? I ask because VFS is used pretty heavily in tests, so having a replacement that's compliant with this new system will be required. And it will be a good stress test that we can expand the backend serialization to other formats in the future.

[hildebrandmw]

After thinking about this for longer than I care to admit, I think we can get rid of enum handling entirely at the trait level. If we consider three cases:

enum UnitEnum {
    A,
    B,
    C,
}

enum InlineStruct {
    A(f32),
    B {
       x: String,
       y: f32,
    }
}

enum Struct {
    A(A),
    B(B),
}

struct A {
    field: f32,
}

struct B {
    x: String,
    y: f32,
}

The serialized representations in JSON could be

{
    "$version": "0.0.0",
    "A": null,
}

{
    $version": "0.0.0",
    "B": {
        "x": "hello world",
        "y": 1,
    }
}

{
    "$version": "0.0.0",
    "A": {
        "$version": "0.0.1",
        "0": 1,
    }
}

For versions 1 and 2, there is just a top level version, which makes sense since the structure of these enums evolves in lockstep with the outer enum. For Version 3, this allows nested objects in enums to have distinct versions naturally.

[suhasjs]

$variant is a little more explicit in how it handles enums. But since most of our enums fall into one of the three variants you described, I've gone ahead and removed $variant handling.

[hildebrandmw]

While I still really like the Light/Heavy idea, if we're honest it's probably overkill and will likely not get implemented ever.

Instead, the main goal I think is to differentiate crucial errors vs "recoverable" errors, where the latter is used for a probing API. In that context, we'd have something like:

struct Error {
    inner: anyhow::Error,
    recoverable: bool,
}

Most of the constructors would imply critical errors, with probing APIs using an additional (_recoverable) API. This keeps normal code simple and the probing API possible and reasonably efficient.

[suhasjs]

It did seem overkill. I've removed it in the latest commit.

[hildebrandmw]

Probably worth adding a

if key.contains("..") || key_as_path.is_absolute() {
   return Err(/* ... */);
}

to keep files inside the directory.

[suhasjs]

Good catch. Added it.

[hildebrandmw]

Should we go ahead and start updating the exiting code to use io::Read/io::Write directly so we don't need this thing?

[suhasjs]

By 'existing code', do you mean all code that uses Storage{Read/Write}Provider traits? Not sure I understood your ask here.

[hildebrandmw]

Most of the code that currently takes Storage{Read/Write}Provider does so by taking the provider and a file name and immediately opening the necessary file. These could all be pretty easily migrated to taking std::io::Read and std::io::Write directly, moving the file open up one level and avoiding the need for this hacky one-off provider.

[hildebrandmw]

In the vein of moving enums to handling themselves, if we add:

impl<'a> Record<'a> {
    fn keys(&self) -> Keys<'_> {
        Keys::new(self.record.key())
    }
}

struct Keys<'a> {
    keys: std::collections::hash_map::Keys<'a, Cow<'a, str>, Value<'a>>,
}

impl<'a> Iterator for Keys<'a> {
    // iterate
}

impl ExactSizeIterator for Keys<'_> {}

Then enums could have a way of inspecting the keys for loading.