Avoid per-row String allocation when hex-encoding digest output (md5, sha1, sha2)

[andygrove]

Is your feature request related to a problem or challenge?

Follow-on from #23766, where @alamb noted the allocation pattern in two places (1, 2).

The digest functions hex-encode their output by allocating one String per row and then copying each of those into the output array:

• datafusion/functions/src/crypto/md5.rs — builds a String per row, collects into a StringViewArray
• datafusion/spark/src/function/hash/sha1.rs — same, into a StringArray
• datafusion/spark/src/function/hash/sha2.rs — same, at eight call sites

All of them go through datafusion_common::utils::hex::encode_bytes, which exists only to return an owned String:

pub fn encode_bytes(bytes: &[u8], case: HexCase) -> String {
    let mut out = Vec::with_capacity(bytes.len() * 2);
    encode_bytes_into(bytes, case, &mut out);
    unsafe { String::from_utf8_unchecked(out) }
}

So each row costs a heap allocation plus a copy into the final array, even though the encoded length is known exactly up front (2 * digest_len, and digest length is fixed per algorithm).

This predates #23766 — that PR consolidated the encoders without changing the calling pattern.

Describe the solution you'd like

Rewrite the callers to encode into a reused buffer and append to a builder, then drop encode_bytes entirely. The shared module already has the non-allocating primitives:

pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>);
pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]) -> Result<()>;

For md5 a StringViewBuilder plus a scratch buffer avoids the intermediate String. For sha1/sha2, whose digests are fixed-size, the output length for the whole array is known in advance, so the same direct value/offset-buffer approach already used by Spark's hex (hex_encode_bytes) and by encode (hex_encode_array) applies.

Removing encode_bytes also shrinks the public API added to datafusion-common in #23766.

Describe alternatives you've considered

Leaving it — the allocation is proportional to row count but small and short-lived, and the digest computation itself dominates. Worth measuring before and after; datafusion/functions/benches/crypto.rs and datafusion/spark/benches/sha2.rs both exist.

Additional context

Encoding::encode_bytes in datafusion/functions/src/encoding/inner.rs also calls it, but only on the scalar path, so that one is not hot.

[Rich-T-kid]

@andygrove can i take this?

[andygrove]

@andygrove can i take this?

Yes, please do. Thanks @Rich-T-kid

[Rich-T-kid]

take