Repository navigation
Conversation
Adds a new `docling-vlm-policy` KFP pipeline that combines Docling's
standard structural parsing with a VLM served via KServe for classifying
and interpreting non-text artifacts (risk matrices, decision trees,
charts, forms, screenshots, diagrams) in policy documents. Each artifact
type gets a specialized prompt producing structured Markdown, assembled
inline with `<!-- artifact: {type} -->` markers.
Includes pipeline definition, interpret_policy_artifacts component,
KServe InferenceService manifest, walkthrough notebook, and README.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add papermill 'parameters' tag to the input files cell and run nbstripout. Also add docling-vlm-policy Python files to Makefile ruff targets. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The 2 failing checks ( |
There was a problem hiding this comment.
I left a few comments on things that would be good to address before merge. The main ones are the missing compiled YAML, the notebook execution in CI, and the local runner secret mount. There are also two smaller suggestions and one question about the test plan.
|
|
||
| ## Installation | ||
|
|
||
| Download the [compiled YAML file](policy_convert_pipeline_compiled.yaml?raw=1) and upload it on the _Import pipeline_ screen in the KFP UI. |
There was a problem hiding this comment.
Thanks for the detailed README! One thing I noticed: the compiled YAML file is not included in the PR, so this link would 404 after merge. The compile workflow also does not cover the new directory yet, so CI would not catch a stale or missing artifact here. Both sibling pipelines commit their compiled YAML and the workflow verifies it stays in sync on every PR.
Suggestion: run python policy_convert_pipeline.py, commit the generated YAML, and add a matrix entry plus a path filter for the new directory in the compile workflow.
There was a problem hiding this comment.
Done — compiled YAML is committed and the compile-kfp workflow now includes a docling-vlm-policy matrix entry with path filters for the new directory.
| "\n", | ||
| "VLM_ENDPOINT_URL = os.getenv(\n", | ||
| " \"VLM_SERVICE_ENDPOINT_URL\",\n", | ||
| " \"http://policy-vlm-predictor.default.svc.cluster.local:8080/v1/chat/completions\",\n", |
There was a problem hiding this comment.
Thanks for the walkthrough notebook, it is very easy to follow! One concern here: this notebook is picked up automatically by the notebook execution tests, which run on push to main. The default flow calls the VLM at this address with no fallback, and the sample PDF contains an image, so the classify step would reach the HTTP call and fail with a connection error in CI.
The existing VLM notebook handles this by defaulting to a local model and only using the remote endpoint when the user opts in. Suggestion: guard the VLM cells so they skip when the endpoint is unreachable, or default to a local model, and move the VLM config into the parameters cell so CI can override it.
There was a problem hiding this comment.
Done — VLM config moved into the parameters cell (papermill-overridable), added a vlm_enabled flag and a connectivity check cell that auto-disables VLM when the endpoint is unreachable. CI will run the notebook cleanly without a GPU endpoint.
| image_export_mode="referenced", | ||
| ) | ||
|
|
||
| interpreter = interpret_policy_artifacts( |
There was a problem hiding this comment.
Thanks for adding the local runner! One issue: the interpreter requires the secret mount and raises a ValueError when it is missing, but the local DockerRunner has no way to mount Kubernetes secrets, so python local_run.py would always fail at this step. The README mentions configuring a VLM secret locally, but there does not seem to be a mechanism that gets it into the container.
make the secret optional with an environment variable fallback for local runs, or document a mechanism that works. Once it runs locally, it would also be nice to add the pipeline to the local runner workflow matrix, which currently omits it.
There was a problem hiding this comment.
Thanks for adding the environment-variable fallback in read_secret — unfortunately I don't think it can take effect in local runs as written. KFP's DockerRunner executes each component in its own container and does not forward host environment variables into it (see kubeflow/pipelines#11030; the local-execution docs also call out that credential/auth mechanisms aren't supported). So when local_run.py reaches interpret_policy_artifacts, neither the /mnt/secrets mount nor the host env vars are visible inside the container, and the step still fails with the same ValueError.
Newer kfp releases accept Docker container run args on DockerRunner (environment, mounts — added in kubeflow/pipelines#12006), so the VLM config could be forwarded explicitly, e.g.:
vlm_env = {
key: os.environ[key]
for key in ("REMOTE_MODEL_ENDPOINT_URL", "REMOTE_MODEL_NAME", "REMOTE_MODEL_API_KEY")
if key in os.environ
}
local.init(runner=local.DockerRunner(environment=vlm_env))Alternatively, a local secrets directory could be mounted via the mounts run arg with vlm_secret_mount_path pointed at it. Good news: the pinned kfp == 2.14.6 already includes the container-run-args support, so no version bump should be needed for this. Once one of these mechanisms is working, it would be great to document the exact steps in the "Local Testing" section of the README.
There was a problem hiding this comment.
On the second point in this thread: docling-vlm-policy is still missing from the local-runner matrix in execute-kfp-localrunners.yml (the matrix still only lists docling-standard and docling-vlm), even though local_run.py was added in this PR. Once the env forwarding above is in place, could we add a matrix entry for it? The "Run local pipeline" step would then need to provide the REMOTE_MODEL_* variables to the container — e.g. exported from a repo secret pointing at a reachable VLM endpoint. If CI can't reach a VLM endpoint, it would help to note that here so the omission reads as intentional rather than an oversight.
There was a problem hiding this comment.
Done — interpret_policy_artifacts now falls back to environment variables when the Kubernetes secret mount path doesn't exist. See alimaredia's follow-up below for the DockerRunner env forwarding gap — addressing that next.
There was a problem hiding this comment.
Good catch — you're right that host env vars don't make it into the DockerRunner container automatically. Updated local_run.py to forward REMOTE_MODEL_* env vars via DockerRunner(environment=vlm_env), and updated the README local testing section with the setup instructions.
There was a problem hiding this comment.
Intentionally omitting from the CI matrix for now — the local runner workflow runs on EC2 GPU instances and there's no VLM endpoint provisioned in CI. Adding it would require standing up a VLM service as part of the workflow, which is a bigger lift. Happy to add it in a follow-up once we have a CI-accessible VLM endpoint or a mock strategy.
| print(f" {doc_name}: saved {policy_md_path.name}", flush=True) | ||
|
|
||
| # Save artifact metadata | ||
| artifacts_json_path = output_path_p / f"{doc_name}_artifacts.json" |
There was a problem hiding this comment.
Nice work on the assembly logic! One small thing: the chunker that runs after this component globs every .json file in the directory and validates it as a DoclingDocument, and the metadata written here does not pass that validation, so every chunking run would log a validation error and a skipped line per document. The real document JSON still chunks fine, so this is noise rather than a break.
filter the _artifacts.json files out of the chunker glob, or write the metadata with a different extension or into a subdirectory.
There was a problem hiding this comment.
Done — artifact metadata is now written to a metadata/ subdirectory so the chunker's *.json glob doesn't pick it up.
| maxReplicas: 1 | ||
| containers: | ||
| - name: kserve-container | ||
| image: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest |
There was a problem hiding this comment.
Thanks for including the manifest! Small suggestion: the image floats on the latest tag, so deployments are not reproducible and could change behavior silently over time. The repo pins versions elsewhere, for example the base images in the constants file. pin a specific tag or digest here.
There was a problem hiding this comment.
Done — pinned to 0.8.5 and added a note in the README about checking the Red Hat container catalog for newer versions.
- Add compiled pipeline YAML and integrate into compile-kfp CI workflow - Guard notebook VLM cells with connectivity check and vlm_enabled flag so CI execution tests pass without a GPU endpoint - Move VLM config into the papermill parameters cell for CI overrides - Fall back to environment variables when Kubernetes secret mount is missing, enabling local runner and non-cluster testing - Write artifact metadata to metadata/ subdirectory to avoid chunker validation errors on non-DoclingDocument JSON files - Pin KServe vLLM image to 0.8.5 instead of floating on latest Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| if not matches: | ||
| print(f" {doc_name}: no images found, copying markdown as-is", flush=True) | ||
| shutil.copy2(md_file, output_path_p / f"{doc_name}_policy.md") | ||
| json_src = input_path_p / f"{doc_name}.json" | ||
| if json_src.exists(): | ||
| shutil.copy2(json_src, output_path_p / f"{doc_name}.json") | ||
| continue |
There was a problem hiding this comment.
A couple of metadata-completeness gaps in this component:
-
No-image documents never get a metadata file. This early-exit path copies the Markdown and DoclingDocument JSON but doesn't write
metadata/{name}_artifacts.json, so downstream consumers can't distinguish "document has zero artifacts" from "interpretation output is missing". -
Skipped images are invisible in the metadata. When
resolve_image_pathfails (the "image not found, skipping" branch just below), the image keeps its rawreference in the assembled Markdown but never appears inartifacts_metadata, sototal_artifactsonly counts successfully interpreted images. That contradicts the README's evaluation guidance, which saystotal_artifacts"should match the number of non-text elements in the source".
Could we always write the metadata file (even an empty one for zero-image documents), and record skipped images — e.g. a skipped array or entries carrying a status field — so artifact coverage is auditable from _artifacts.json alone?
There was a problem hiding this comment.
Both addressed:
- No-image documents now always get a metadata file with
total_images: 0,total_artifacts: 0, emptyskippedandartifactsarrays. - Skipped images are now recorded in a
skippedarray withindex,source_image, andreasonfields. Addedtotal_images(count of all image references found) alongsidetotal_artifacts(count successfully interpreted) so coverage is auditable from the metadata alone.
- Forward VLM env vars into DockerRunner via environment param so local_run.py actually works without Kubernetes secrets - Write metadata file for no-image documents so downstream consumers can distinguish "zero artifacts" from "output missing" - Track skipped images in metadata with reason field, add total_images count so artifact coverage is auditable from _artifacts.json alone - Update README local testing section with env var setup instructions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
commit_message_template was removed by Mergify upstream. Migrate to the declarative commit_message_format setting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Summary
docling-vlm-policyKFP pipeline that combines Docling's standard structural parsing with a VLM (served via KServe) for classifying and interpreting non-text artifacts in policy documentsinterpret_policy_artifactscomponent classifies each extracted image (chart, risk matrix, decision tree, form, screenshot, diagram, other) and interprets it with a type-specific structured prompt, assembling the result inline with<!-- artifact: {type} -->markersregistry.redhat.io/rhaiis/vllm-cuda-rhel9), walkthrough notebook, and full README with architecture, deployment, and evaluation methodologyNew files
kubeflow-pipelines/docling-vlm-policy/— pipeline definition, components, local runner, KServe manifest, READMEnotebooks/use-cases/policy-document-conversion-vlm.ipynb— step-by-step walkthroughDesign
Uses the standard Docling pipeline (not VLM pipeline) for structural conversion with
image_export_mode="referenced"to extract images as PNGs. A separateinterpret_policy_artifactsstep sends each image to an external VLM for classification and type-specific interpretation, then replaces image references in the Markdown with structured interpretations. Reusesimport_pdfs,create_pdf_splits,download_docling_models,docling_convert_standard, anddocling_chunkfrom existing pipelines.Test plan
python policy_convert_pipeline.pyproduces valid YAML/v1/chat/completionsresponds_policy.mdoutput contains<!-- artifact: type -->markers with structured content_artifacts.jsonmetadata matches image count and typesnum_splits=5for parallel processing🤖 Generated with Claude Code