Skip to content

Add policy document conversion pipeline with VLM artifact interpretation - #128

Open
alinaryan wants to merge 6 commits into
opendatahub-io:mainfrom
alinaryan:docling-vlm-policy-pipeline
Open

alinaryan wants to merge 6 commits into
opendatahub-io:mainfrom
alinaryan:docling-vlm-policy-pipeline

Conversation

@alinaryan

Copy link
Copy Markdown
Contributor

Summary

  • Adds a new docling-vlm-policy KFP pipeline that combines Docling's standard structural parsing with a VLM (served via KServe) for classifying and interpreting non-text artifacts in policy documents
  • New interpret_policy_artifacts component classifies each extracted image (chart, risk matrix, decision tree, form, screenshot, diagram, other) and interprets it with a type-specific structured prompt, assembling the result inline with <!-- artifact: {type} --> markers
  • Includes KServe InferenceService manifest for Granite Vision via vLLM (registry.redhat.io/rhaiis/vllm-cuda-rhel9), walkthrough notebook, and full README with architecture, deployment, and evaluation methodology

New files

  • kubeflow-pipelines/docling-vlm-policy/ — pipeline definition, components, local runner, KServe manifest, README
  • notebooks/use-cases/policy-document-conversion-vlm.ipynb — step-by-step walkthrough

Design

Uses the standard Docling pipeline (not VLM pipeline) for structural conversion with image_export_mode="referenced" to extract images as PNGs. A separate interpret_policy_artifacts step sends each image to an external VLM for classification and type-specific interpretation, then replaces image references in the Markdown with structured interpretations. Reuses import_pdfs, create_pdf_splits, download_docling_models, docling_convert_standard, and docling_chunk from existing pipelines.

Test plan

  • Compile pipeline: python policy_convert_pipeline.py produces valid YAML
  • Deploy VLM InferenceService and verify /v1/chat/completions responds
  • Run pipeline on sample PDFs with images and verify artifact classification + interpretation output
  • Verify _policy.md output contains <!-- artifact: type --> markers with structured content
  • Verify _artifacts.json metadata matches image count and types
  • Test with 200+ page corpus using num_splits=5 for parallel processing
  • Run walkthrough notebook end-to-end

🤖 Generated with Claude Code

alinaryan and others added 2 commits October 15, 2026 09:45
Adds a new `docling-vlm-policy` KFP pipeline that combines Docling's
standard structural parsing with a VLM served via KServe for classifying
and interpreting non-text artifacts (risk matrices, decision trees,
charts, forms, screenshots, diagrams) in policy documents. Each artifact
type gets a specialized prompt producing structured Markdown, assembled
inline with `<!-- artifact: {type} -->` markers.

Includes pipeline definition, interpret_policy_artifacts component,
KServe InferenceService manifest, walkthrough notebook, and README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add papermill 'parameters' tag to the input files cell and run
nbstripout. Also add docling-vlm-policy Python files to Makefile
ruff targets.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@alinaryan
alinaryan requested a review from a team as a code owner October 15, 2026 13:50
@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML (base), Organization UI (inherited)
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3042347e-b83e-439d-adf1-8766cfe62308

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@alinaryan

Copy link
Copy Markdown
Contributor Author

The 2 failing checks (launch-ec2-runner, stop-ec2-runner) are an existing CI issue — the if: github.repository == 'opendatahub-io/data-processing' guard doesn't prevent the jobs from being created, and they fail on fork PRs because the AWS OIDC secrets aren't available. This affects all fork PRs, not just this one. All code validation checks (Python formatting, notebook validation, notebook parameters) pass.

@RobuRishabh RobuRishabh left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few comments on things that would be good to address before merge. The main ones are the missing compiled YAML, the notebook execution in CI, and the local runner secret mount. There are also two smaller suggestions and one question about the test plan.


## Installation

Download the [compiled YAML file](policy_convert_pipeline_compiled.yaml?raw=1) and upload it on the _Import pipeline_ screen in the KFP UI.

@RobuRishabh RobuRishabh Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed README! One thing I noticed: the compiled YAML file is not included in the PR, so this link would 404 after merge. The compile workflow also does not cover the new directory yet, so CI would not catch a stale or missing artifact here. Both sibling pipelines commit their compiled YAML and the workflow verifies it stays in sync on every PR.
Suggestion: run python policy_convert_pipeline.py, commit the generated YAML, and add a matrix entry plus a path filter for the new directory in the compile workflow.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — compiled YAML is committed and the compile-kfp workflow now includes a docling-vlm-policy matrix entry with path filters for the new directory.

"\n",
"VLM_ENDPOINT_URL = os.getenv(\n",
" \"VLM_SERVICE_ENDPOINT_URL\",\n",
" \"http://policy-vlm-predictor.default.svc.cluster.local:8080/v1/chat/completions\",\n",

@RobuRishabh RobuRishabh Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the walkthrough notebook, it is very easy to follow! One concern here: this notebook is picked up automatically by the notebook execution tests, which run on push to main. The default flow calls the VLM at this address with no fallback, and the sample PDF contains an image, so the classify step would reach the HTTP call and fail with a connection error in CI.
The existing VLM notebook handles this by defaulting to a local model and only using the remote endpoint when the user opts in. Suggestion: guard the VLM cells so they skip when the endpoint is unreachable, or default to a local model, and move the VLM config into the parameters cell so CI can override it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — VLM config moved into the parameters cell (papermill-overridable), added a vlm_enabled flag and a connectivity check cell that auto-disables VLM when the endpoint is unreachable. CI will run the notebook cleanly without a GPU endpoint.

image_export_mode="referenced",
)

interpreter = interpret_policy_artifacts(

@RobuRishabh RobuRishabh Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the local runner! One issue: the interpreter requires the secret mount and raises a ValueError when it is missing, but the local DockerRunner has no way to mount Kubernetes secrets, so python local_run.py would always fail at this step. The README mentions configuring a VLM secret locally, but there does not seem to be a mechanism that gets it into the container.
make the secret optional with an environment variable fallback for local runs, or document a mechanism that works. Once it runs locally, it would also be nice to add the pipeline to the local runner workflow matrix, which currently omits it.

@alimaredia alimaredia Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the environment-variable fallback in read_secret — unfortunately I don't think it can take effect in local runs as written. KFP's DockerRunner executes each component in its own container and does not forward host environment variables into it (see kubeflow/pipelines#11030; the local-execution docs also call out that credential/auth mechanisms aren't supported). So when local_run.py reaches interpret_policy_artifacts, neither the /mnt/secrets mount nor the host env vars are visible inside the container, and the step still fails with the same ValueError.

Newer kfp releases accept Docker container run args on DockerRunner (environment, mounts — added in kubeflow/pipelines#12006), so the VLM config could be forwarded explicitly, e.g.:

vlm_env = {
    key: os.environ[key]
    for key in ("REMOTE_MODEL_ENDPOINT_URL", "REMOTE_MODEL_NAME", "REMOTE_MODEL_API_KEY")
    if key in os.environ
}
local.init(runner=local.DockerRunner(environment=vlm_env))

Alternatively, a local secrets directory could be mounted via the mounts run arg with vlm_secret_mount_path pointed at it. Good news: the pinned kfp == 2.14.6 already includes the container-run-args support, so no version bump should be needed for this. Once one of these mechanisms is working, it would be great to document the exact steps in the "Local Testing" section of the README.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the second point in this thread: docling-vlm-policy is still missing from the local-runner matrix in execute-kfp-localrunners.yml (the matrix still only lists docling-standard and docling-vlm), even though local_run.py was added in this PR. Once the env forwarding above is in place, could we add a matrix entry for it? The "Run local pipeline" step would then need to provide the REMOTE_MODEL_* variables to the container — e.g. exported from a repo secret pointing at a reachable VLM endpoint. If CI can't reach a VLM endpoint, it would help to note that here so the omission reads as intentional rather than an oversight.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — interpret_policy_artifacts now falls back to environment variables when the Kubernetes secret mount path doesn't exist. See alimaredia's follow-up below for the DockerRunner env forwarding gap — addressing that next.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — you're right that host env vars don't make it into the DockerRunner container automatically. Updated local_run.py to forward REMOTE_MODEL_* env vars via DockerRunner(environment=vlm_env), and updated the README local testing section with the setup instructions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentionally omitting from the CI matrix for now — the local runner workflow runs on EC2 GPU instances and there's no VLM endpoint provisioned in CI. Adding it would require standing up a VLM service as part of the workflow, which is a bigger lift. Happy to add it in a follow-up once we have a CI-accessible VLM endpoint or a mock strategy.

print(f" {doc_name}: saved {policy_md_path.name}", flush=True)

# Save artifact metadata
artifacts_json_path = output_path_p / f"{doc_name}_artifacts.json"

@RobuRishabh RobuRishabh Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on the assembly logic! One small thing: the chunker that runs after this component globs every .json file in the directory and validates it as a DoclingDocument, and the metadata written here does not pass that validation, so every chunking run would log a validation error and a skipped line per document. The real document JSON still chunks fine, so this is noise rather than a break.
filter the _artifacts.json files out of the chunker glob, or write the metadata with a different extension or into a subdirectory.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — artifact metadata is now written to a metadata/ subdirectory so the chunker's *.json glob doesn't pick it up.

maxReplicas: 1
containers:
- name: kserve-container
image: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest

@RobuRishabh RobuRishabh Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for including the manifest! Small suggestion: the image floats on the latest tag, so deployments are not reproducible and could change behavior silently over time. The repo pins versions elsewhere, for example the base images in the constants file. pin a specific tag or digest here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — pinned to 0.8.5 and added a note in the README about checking the Red Hat container catalog for newer versions.

- Add compiled pipeline YAML and integrate into compile-kfp CI workflow
- Guard notebook VLM cells with connectivity check and vlm_enabled flag
  so CI execution tests pass without a GPU endpoint
- Move VLM config into the papermill parameters cell for CI overrides
- Fall back to environment variables when Kubernetes secret mount is
  missing, enabling local runner and non-cluster testing
- Write artifact metadata to metadata/ subdirectory to avoid chunker
  validation errors on non-DoclingDocument JSON files
- Pin KServe vLLM image to 0.8.5 instead of floating on latest

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment on lines +241 to +247
if not matches:
print(f" {doc_name}: no images found, copying markdown as-is", flush=True)
shutil.copy2(md_file, output_path_p / f"{doc_name}_policy.md")
json_src = input_path_p / f"{doc_name}.json"
if json_src.exists():
shutil.copy2(json_src, output_path_p / f"{doc_name}.json")
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of metadata-completeness gaps in this component:

  1. No-image documents never get a metadata file. This early-exit path copies the Markdown and DoclingDocument JSON but doesn't write metadata/{name}_artifacts.json, so downstream consumers can't distinguish "document has zero artifacts" from "interpretation output is missing".

  2. Skipped images are invisible in the metadata. When resolve_image_path fails (the "image not found, skipping" branch just below), the image keeps its raw ![...](...) reference in the assembled Markdown but never appears in artifacts_metadata, so total_artifacts only counts successfully interpreted images. That contradicts the README's evaluation guidance, which says total_artifacts "should match the number of non-text elements in the source".

Could we always write the metadata file (even an empty one for zero-image documents), and record skipped images — e.g. a skipped array or entries carrying a status field — so artifact coverage is auditable from _artifacts.json alone?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both addressed:

  1. No-image documents now always get a metadata file with total_images: 0, total_artifacts: 0, empty skipped and artifacts arrays.
  2. Skipped images are now recorded in a skipped array with index, source_image, and reason fields. Added total_images (count of all image references found) alongside total_artifacts (count successfully interpreted) so coverage is auditable from the metadata alone.

alinaryan and others added 2 commits November 11, 2026 13:30
- Forward VLM env vars into DockerRunner via environment param so
  local_run.py actually works without Kubernetes secrets
- Write metadata file for no-image documents so downstream consumers
  can distinguish "zero artifacts" from "output missing"
- Track skipped images in metadata with reason field, add total_images
  count so artifact coverage is auditable from _artifacts.json alone
- Update README local testing section with env var setup instructions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
commit_message_template was removed by Mergify upstream. Migrate to
the declarative commit_message_format setting.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants