feat(encryption) [1/N] Support encryption: Add crypto for AES-GCM

[xanderbailey]

Add Core Encryption Primitives for Iceberg Encryption Support.

Part of #2034

Summary

This PR introduces the foundational cryptographic primitives needed for implementing encryption in iceberg-rust, providing AES-GCM encryption operations that match the Java implementation's behavior and data format.

Motivation

Iceberg's Java implementation supports table-level encryption to protect sensitive data at rest. To achieve feature parity and ensure interoperability between Java and Rust implementations, we need to build encryption support from the ground up. This PR provides the core cryptographic operations that will serve as the foundation for the complete encryption feature.

Changes

New Module: encryption

Added a new encryption module with core AES-GCM cryptographic operations:

• encryption/crypto.rs - Core encryption implementation
  • EncryptionAlgorithm enum supporting AES-128-GCM as this is the only algorithm currently supported in arrow parquet
  • SecureKey struct with automatic memory zeroization for security
  • AesGcmEncryptor providing encrypt/decrypt operations with AAD support

Key Features

1. Java-Compatible Format: Ciphertext format matches Java's implementation exactly:
   [12-byte nonce][encrypted data][16-byte GCM authentication tag]
2. This ensures files encrypted by Java can be decrypted by Rust and vice versa.
3. Secure Key Handling: Uses the zeroize crate to automatically clear encryption keys from memory when dropped, preventing key material from lingering in memory.
4. Additional Authenticated Data (AAD): Full support for AAD to ensure integrity of associated metadata that isn't encrypted.
5. Comprehensive Testing: 8 tests covering:
   - Round-trip encryption/decryption for both AES-128 and AES-256
   - AAD validation
   - Empty plaintext handling
   - Tamper detection
   - Format compatibility verification

Dependencies Added

• aes-gcm = "0.10" - Industry-standard AES-GCM implementation
• zeroize = "1.7" - Secure memory cleanup for encryption keys

Compatibility

This implementation directly corresponds to Java's https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/encryption/Ciphers.java:

Java Class	Rust Implementation
Ciphers.AesGcmEncryptor	AesGcmEncryptor::encrypt()
Ciphers.AesGcmDecryptor	AesGcmEncryptor::decrypt()
EncryptionAlgorithm.AES_GCM	EncryptionAlgorithm::Aes128Gcm

Testing

Future Work

This PR is the first in a series to implement full encryption support. Upcoming PRs will add:

1. Table properties for encryption configuration
2. Key management interfaces (KeyManagementClient trait)
3. EncryptionManager implementation
4. Native Parquet encryption integration
5. AWS KMS support
6. Integration with Table and FileIO

Review Notes

• This PR is intentionally minimal and self-contained
• No existing code paths are modified - this is purely additive
• The module is public but won't be used until future PRs wire it up
• Format compatibility with Java has been prioritized to ensure interoperability

Which issue does this PR close?

• Closes #. Encryption - Add AES-GCM encryption utilities #2035

What changes are included in this PR?

Are these changes tested?

Yes

[mbutrovich]

This is awesome! I will take a look since I did a lot of the PME support in Comet. Regarding AES-256-GCM support, last I looked Arrow-rs' Parquet reader only supported 128. Is that not the case anymore?

[xanderbailey]

You are correct by the looks of it! https://github.com/apache/arrow-rs/blob/main/parquet/src/encryption/ciphers.rs

[hsiang-c]

@xanderbailey Thank you for the great work on encryption support!

Regarding arror-rs, I removed the hardcoded AES-128 constraint w/ apache/arrow-rs#9203, hope that helps a bit. Because we're using PME with AES-256

cc @mbutrovich

[xanderbailey]

This helps a bunch thanks!

[xanderbailey]

@mbutrovich are you still okay to review this? Keen to start getting stuff merged!

[mbutrovich]

This foundation looks solid to me. We'll need a committer to approve, but based on previous Parquet Modular Encryption work this is off to a good start. Thank you for driving this work, @xanderbailey!

[mbutrovich]

We're trying to get 256 added to Arrow-rs, this approach should work with that?

[xanderbailey]

Correct, the crate has both! Designed to be extended once the arrow PR merges

[jdockerty]

once the arrow PR merges

Not a huge deal, but I wonder whether you can save a bit of extra review churn by filling that part in already and having the pre-existing FeatureUnsupported error in this PR?

iceberg-rust/crates/iceberg/src/error.rs

Lines 59 to 62 in 854f66d

	/// Iceberg feature is not supported.
	///
	/// This error is returned when given iceberg feature is not supported.
	FeatureUnsupported,

As I say, this is also probably quite easy to do in a follow-up too once it does merge. So not a huge deal 👍

[blackmwk]

Do we really need this enum? I think iceberg currently only support AES_GCM, and the enum is actully not used anyware.

[xanderbailey]

We will support 256 bit as soon is we get support in the parquet reader / writer so I wanted to trivially be able to extend this.

[blackmwk]

Could we remove this enum for now? I checked java's source code and only find one corresponding enum: https://github.com/apache/iceberg/blob/bec6793af822cb791fa5975b41b8936631556c46/core/src/main/java/org/apache/iceberg/encryption/NativeFileCryptoParameters.java#L32

It's already marked as deprecated. I'm not sure if we actually need this enum.

[mbutrovich]

Iceberg spec defers data file encryption for Parquet to its spec, which supports 128, 192, and 256 bit.

https://parquet.apache.org/docs/file-format/data-pages/encryption/

Iceberg metadata files can use AES GCM 128, 192, and 256 bit.

https://iceberg.apache.org/gcm-stream-spec/#goals

We should keep the enum, or expect to add something similar back in the future (maybe in the encryption RFC?). It seems like we should anticipate a possible API break in the future and have the enum now.

[blackmwk]

I'm still quite confused because I don't see the usage of EncryptionAlgorithm in java. If we have to add this, I'm thinking sth as following:

pub enum AesKeySize {
   Bits128  = 128,
   Bits192  = 192,
   Bits256 = 256
}

And this could be used in following.

[blackmwk]

Please correct me if I'm wrong, and point a source where it's used in java codebase.

[xanderbailey]

It's not used in java but java's Cipher class can handle just the length because the underlying library is happy to accept a key length which is not true for us so we need to dispatch somewhere, I guess my design had this dispatch on creation but your suggestion is to do it at encrypt / decrypt time which is totally fine also. Happy to make this change. Thanks for iterating with me.

[mbutrovich]

Same question about 256 for these following functions.

[mbutrovich]

Good, clears it.

[xanderbailey]

Tried to follow best practices here

[xanderbailey]

Thanks for the review! I'm new to contributing to iceberg, what's the convention for getting a committer to review?

[mbutrovich]

Thanks for the review! I'm new to contributing to iceberg, what's the convention for getting a committer to review?

We need an Iceberg committer, most likely one of the folks working more closely on the Iceberg-rust repo (@Xuanwo and @blackmwk). However, with Lunar New Year I think they'll be slow to respond for a little bit.

[mbutrovich]

I think is ready for review from a committer @Xuanwo @blackmwk. @xanderbailey has a number of PRs queued up to get encryption support going in iceberg-rust.

[blackmwk]

Thanks @xanderbailey and @mbutrovich for this pr, I'll take a look recently.

[blackmwk]

Hi, @xanderbailey Iceberg's encryption support is relative large piece. Would you mind to write an rfc to describe the overall design and the split them into small prs to do it? Here is a good example of this process: #1885

[xanderbailey]

Happy to write an RFC. Will work on that today.

[xanderbailey]

RFC: #2183

[jdockerty]

This looks really nice and it is easy to follow. Great stuff, it is definitely ready for maintainer review 👍

I see there was an ask on writing up an RFC too. I think it is worth including that RFC link either directly in this PR's description and/or in the encryption/mod.rs so that it is easy to discover for someone browsing around trying to understand the design decisions.

[jdockerty]

Love this 💯

[blackmwk]

+1

[jdockerty]

once the arrow PR merges

Not a huge deal, but I wonder whether you can save a bit of extra review churn by filling that part in already and having the pre-existing FeatureUnsupported error in this PR?

iceberg-rust/crates/iceberg/src/error.rs

Lines 59 to 62 in 854f66d

	/// Iceberg feature is not supported.
	///
	/// This error is returned when given iceberg feature is not supported.
	FeatureUnsupported,

As I say, this is also probably quite easy to do in a follow-up too once it does merge. So not a huge deal 👍

[jdockerty]

This might be better mapped to the FeatureUnsupported error, considering the message is also about an unsupported algorithm. Rather than invalid data (although I do think DataInvalid works anyway)

[xanderbailey]

Yeah I wasn't sure about this one, it could be an algorithm unsupported by the spec or an algorithm unsupported by iceberg-rs and I wasn't sure how to distinguish these.

[xanderbailey]

@blackmwk how would you feel about starting to get some parts of the implementation merged whilst we work out some final details of the RFC?

[blackmwk]

@blackmwk how would you feel about starting to get some parts of the implementation merged whilst we work out some final details of the RFC?

Hi, @xanderbailey Yeah, we could move on with small part that we reached consensus.

[blackmwk]

Thanks @xanderbailey for this pr, left some comments and questions. I have to admit that I'm not an expert in encryption, so hopefully you could resolve my questions.

[blackmwk]

+1

[blackmwk]

Do we really need this enum? I think iceberg currently only support AES_GCM, and the enum is actully not used anyware.

[blackmwk]

It's a little odd. It's called AesGcmEncryptor, but it has enum for support inside. I checked java's code, and I think currently iceberg only support AesGcm? I'm thinking we only need need a struct
AesGcmCipher, which could be used for do encryption/decrpytion.

[xanderbailey]

Yes, the additional option is to use 256 which is still AES-GCM but different length keys.

[xanderbailey]

I can add this once apache/arrow-rs#9203 has merged and we bump arrow.

[blackmwk]

What I want to say is that other encryption algorithms (e.g. aes gcm 256) are format specific, and currently in iceberg itself, we only need to keep AesGcmEncryptor. The support for format specific algorithm should be in format specific module like arrow.
In my opinion, in this pr we only need to keep AesGcmEncryptor, and remove other components.

[xanderbailey]

The algorithm selection isn't just for Parquet data files - it's also needed for encrypting metadata files (manifests, manifest lists) at the core Iceberg layer, which never goes through the arrow/format module. The encryption.data-key-length table property controls DEK generation for all encrypted files uniformly, so the core crate needs to know whether it's 128 or 256.

[xanderbailey]

Sorry if I have misunderstood your point here!

[blackmwk]

I still don't quite understand? Will there be huge difference for different key length? Will not such thing be more concise?

pub struct AesGcmEncryptor {
  /// From table property
   key_length: usize
}

[xanderbailey]

The aes_gcm crate uses distinct types for Aes128Gcm and Aes256Gcm - they're different generic instantiations, not runtime-configurable. A key_length: usize field can't select between them at the type level, so we'd still need branching logic somewhere. The CipherImpl enum resolves this dispatch once at construction time rather than on every encrypt/decrypt call which seems more idiomatic to me?

[blackmwk]

Sorry, I'm not quite convinced, in aes_gcm both and Aes128Gsm and Aes256Gsm are just type aliases. With the enum in https://github.com/apache/iceberg-rust/pull/2026/changes#r2957736592, we could do following:

match AesKeySize {
  Bit128 => encrypt<Aes, Nounce128>(...)
Bit256 => encrypt<Aes, Nounce256>(...)
}
fn encrypt<Aes, NouunceSize> (...) {
    ...
}

[blackmwk]

Could we remove this enum for now? I checked java's source code and only find one corresponding enum: https://github.com/apache/iceberg/blob/bec6793af822cb791fa5975b41b8936631556c46/core/src/main/java/org/apache/iceberg/encryption/NativeFileCryptoParameters.java#L32

It's already marked as deprecated. I'm not sure if we actually need this enum.

[blackmwk]

I still don't quite understand? Will there be huge difference for different key length? Will not such thing be more concise?

pub struct AesGcmEncryptor {
  /// From table property
   key_length: usize
}

[blackmwk]

Suggested change

	pub struct AesGcmEncryptor {
	pub struct AesGcmCipher {

Approved by mistake.

[xanderbailey]

The aes_gcm crate uses distinct types for Aes128Gcm and Aes256Gcm - they're different generic instantiations, not runtime-configurable. A key_length: usize field can't select between them at the type level, so we'd still need branching logic somewhere. The CipherImpl enum resolves this dispatch once at construction time rather than on every encrypt/decrypt call which seems more idiomatic to me?

[blackmwk]

Thanks @xanderbailey for this pr! Generally LGTM, just several minor suggestions.

[blackmwk]

Suggested change

	pub struct SensitiveBytes(Zeroizing<Box<[u8]>>);
	struct SensitiveBytes(Zeroizing<Box<[u8]>>);

I think we are expecting to expose SecureKey as public api, and we should hide this?

[xanderbailey]

092c25a

[blackmwk]

Thanks @xanderbailey for this pr!