Add large payload utilities

[SarthakShrivastav-a]

Summary

• add utils.large_payload helpers for publishing application payloads that may exceed practical data-packet size limits
• use a compact descriptor on a data topic, with inline base64 for small payloads and stream_bytes fallback for larger payloads
• add receiver-side helpers for descriptor parsing, streamed-byte size checks, and SHA-256 verification
• add a generic example under examples/other and focused unit coverage

Refs #6906

Testing

• python -m ruff check --fix livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py

[CLAassistant]

All committers have signed the CLA.

[SarthakShrivastav-a]

Addressed the review comments in the latest push:

• avoided base64/JSON construction for the inline descriptor when inline transfer is disabled or cannot fit by estimating the descriptor size first
• added public docstrings for descriptor serialization/deserialization helpers
• made stream reads bounded by default and updated the example receiver to pass an explicit max byte limit
• added regression coverage for inline encoding bypass, stream-size enforcement, configurable stream limits, and pre-read expected-size rejection

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py

[SarthakShrivastav-a]

Addressed the latest review comment in the latest push.

The inline transfer decision now clamps to min(max_inline_bytes, max_descriptor_bytes) for both the estimated descriptor gate and the final encoded descriptor check, so inline payload descriptors cannot exceed the configured descriptor/data-message cap. Added regression coverage for the case where max_inline_bytes is higher than max_descriptor_bytes and the payload should fall back to byte streams.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check

[SarthakShrivastav-a]

Addressed the latest example review comment in the latest push.

The example receiver now keeps strong references to in-flight byte-stream read tasks, discards them when complete, exposes cleanup via aclose(), and logs stream-read failures so validation errors are visible instead of being dropped by an unobserved task.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py examples/other/large-payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check

[SarthakShrivastav-a]

Addressed the parser review comment in the latest push.

parse_large_payload_descriptor now wraps invalid UTF-8 descriptor bytes in LargePayloadError, matching the documented public error contract. Added regression coverage for corrupt descriptor bytes.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py examples/other/large-payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check

[SarthakShrivastav-a]

Addressed the content-type validation review comment in the latest push.

publish_large_payload now rejects an empty content_type before constructing descriptors, so the publisher cannot emit descriptors that its own parser rejects. Added regression coverage for that validation path.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py examples/other/large-payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check

[SarthakShrivastav-a]

Addressed the latest two review comments in the latest push.

The byte-stream publish path now abort-closes the writer with a reason when write() fails, suppressing close cleanup errors so the original write failure is preserved. Payload checksum validation now uses hmac.compare_digest. Added regression coverage for the failed write/abort path.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py examples/other/large-payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check

[SarthakShrivastav-a]

Addressed the latest checksum and stream-validation review comments in the latest push.

read_large_payload_stream now requires expected size and SHA-256 metadata by default, with an explicit opt-out for trusted or separately validated streams. SHA-256 text is normalized to lowercase hex, rejected if malformed, and then compared with hmac.compare_digest. Added regression coverage for missing metadata, malformed checksum text, uppercase checksum text, and the explicit max-byte opt-out path.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py examples/other/large-payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check

[SarthakShrivastav-a]

Addressed the latest example review comment in the latest push.

The receiver example now uses an explicit expected_size is not None check when converting the stream size attribute, which keeps zero-length streams unambiguous and avoids treating metadata parsing as a truthiness check.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py examples/other/large-payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check

[SarthakShrivastav-a]

Addressed the latest descriptor-location review comment in the latest push.

LargePayloadDescriptor.from_dict now rejects byte_stream descriptors that omit stream_topic or stream_name, and also rejects inline descriptors that omit the data field. Added parser regression coverage for both malformed descriptor shapes.

Validation:

• python -m ruff check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• python -m ruff format --check livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py livekit-agents/livekit/agents/utils/__init__.py examples/other/large-payload.py
• PYTHONPATH=livekit-agents python -m pytest tests/test_large_payload.py --unit
• MYPYPATH=livekit-agents python -m mypy --explicit-package-bases livekit-agents/livekit/agents/utils/large_payload.py examples/other/large-payload.py
• python -m py_compile livekit-agents/livekit/agents/utils/large_payload.py tests/test_large_payload.py examples/other/large-payload.py
• git diff --check