Skip to content

feat: add raw log ingestion helpers#101

Merged
beinan merged 1 commit into
mainfrom
codex/issue-97-raw-ingestion
Jun 24, 2026
Merged

feat: add raw log ingestion helpers#101
beinan merged 1 commit into
mainfrom
codex/issue-97-raw-ingestion

Conversation

@dcfocus

@dcfocus dcfocus commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add Python helpers for raw interaction-log ingestion from mapped row iterables, JSONL, and OpenAI-style messages
  • preserve raw provenance under metadata and allow caller-supplied run_id / created_at on local add, add_many, and upsert paths
  • add focused tests for provenance mapping, streaming JSONL batches, idempotent upsert ingestion, and chat message ingestion

Testing

  • cargo fmt --manifest-path python/Cargo.toml
  • cargo fmt --manifest-path python/Cargo.toml -- --check
  • env PROTOC=/home/user/data-iq/.venv/lib/python3.12/site-packages/torch/bin/protoc PROTOC_INCLUDE=/home/user/data-iq/.venv/lib/python3.12/site-packages/grpc_tools/_proto uv run --project python --extra dev ruff check python/python/lance_context/api.py python/tests/test_ingestion.py
  • env PROTOC=/home/user/data-iq/.venv/lib/python3.12/site-packages/torch/bin/protoc PROTOC_INCLUDE=/home/user/data-iq/.venv/lib/python3.12/site-packages/grpc_tools/_proto uv run --project python --extra tests pytest python/tests/test_ingestion.py python/tests/test_add_many.py python/tests/test_state_metadata.py python/tests/test_external_id.py
  • env PROTOC=/home/user/data-iq/.venv/lib/python3.12/site-packages/torch/bin/protoc PROTOC_INCLUDE=/home/user/data-iq/.venv/lib/python3.12/site-packages/grpc_tools/_proto uv run --project python --extra dev pyright python/python/lance_context/api.py python/python/lance_context/init.py
  • git diff --check

Notes

  • The repo PR helper script was attempted but is stale for the current layout: it fails on missing manifest path rust/lance-context/Cargo.toml before running checks.
  • Full-repo pyright still reports existing unrelated errors in examples/tests, so this PR uses a scoped pyright check for the touched public API files.

Closes #97

@dcfocus dcfocus marked this pull request as ready for review June 24, 2026 14:04
@beinan beinan merged commit da0088f into main Jun 24, 2026
5 checks passed
@dcfocus dcfocus deleted the codex/issue-97-raw-ingestion branch June 27, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingest raw interaction logs into lance-context records (batch + streaming)

2 participants