Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 69 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ libc = "0.2.183"
predicates = "3.1.4"
process-wrap = { version = "9.1.0", default-features = false, features = ["job-object", "kill-on-drop", "tokio1"] }
reqwest = { version = "0.13.4", default-features = false, features = ["brotli", "charset", "gzip", "http2", "rustls-no-provider", "stream", "system-proxy", "zstd"] }
quick-xml = "0.39.4"
ring = "0.17.14"
rustls = { version = "0.23.42", default-features = false, features = ["ring", "std", "tls12"] }
semver = "1.0.28"
Expand All @@ -44,6 +45,7 @@ unicode-segmentation = "1.13.3"
url = { version = "2.5.8", features = ["serde"] }
wiremock = "0.6.5"
windows = "0.62.2"
zip = { version = "2.4.2", default-features = false, features = ["deflate"] }

[workspace.lints.rust]
unsafe_code = "forbid"
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ Opsail is a modular native toolkit that gives software agents small, composable,

## Core capabilities

### Read HTML
### Read HTML and XLSX

`opsail read` turns static HTML or a browser-rendered DOM into readable Markdown, sanitized HTML, or versioned JSON. It accepts URLs, files, stdin, an Opsail-owned isolated Chrome process, or an explicitly borrowed CDP endpoint.
`opsail read` turns static HTML or a browser-rendered DOM into readable Markdown, sanitized HTML, or versioned JSON. It also reads bounded sparse ranges from local XLSX workbooks. HTML accepts URLs, files, stdin, an Opsail-owned isolated Chrome process, or an explicitly borrowed CDP endpoint.

```sh
opsail read https://example.com/article
opsail read https://example.com/app --launch
opsail read ./book.xlsx --max-bytes 67108864 --range 'Summary!A1:H30' --format json
```

See [`opsail-read`](https://github.com/lencx/opsail/blob/main/crates/opsail-read/README.md) for acquisition, extraction, result contracts, and Rust APIs. See [`opsail-chrome`](https://github.com/lencx/opsail/blob/main/crates/opsail-chrome/README.md) for Chrome discovery, owned launch, borrowed CDP, navigation, and rendered DOM capture.
Expand Down
5 changes: 3 additions & 2 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ Opsail 是一个模块化原生工具集,通过统一的命令行入口,为

## 核心能力

### 读取 HTML
### 读取 HTML 与 XLSX

`opsail read` 可以将静态 HTML 或浏览器渲染后的 DOM 转换为易读的 Markdown、经过清理的 HTML 或带版本的 JSON。输入可以来自 URL、文件、stdin、由 Opsail 启动的隔离 Chrome,或显式借用的 CDP 端点。
`opsail read` 可以将静态 HTML 或浏览器渲染后的 DOM 转换为易读的 Markdown、经过清理的 HTML 或带版本的 JSON,也可以稀疏读取本地 XLSX 的受控范围。HTML 输入可以来自 URL、文件、stdin、由 Opsail 启动的隔离 Chrome,或显式借用的 CDP 端点。

```sh
opsail read https://example.com/article
opsail read https://example.com/app --launch
opsail read ./book.xlsx --max-bytes 67108864 --range 'Summary!A1:H30' --format json
```

内容获取、正文提取、结果契约和 Rust API 请参阅 [`opsail-read`](https://github.com/lencx/opsail/blob/main/crates/opsail-read/README.md);Chrome 发现、自有启动、借用 CDP、页面导航和渲染 DOM 捕获请参阅 [`opsail-chrome`](https://github.com/lencx/opsail/blob/main/crates/opsail-chrome/README.md)。
Expand Down
3 changes: 3 additions & 0 deletions crates/opsail-read/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ dom_smoothie.workspace = true
encoding_rs.workspace = true
futures-util.workspace = true
opsail-chrome = { path = "../opsail-chrome", version = "0.1.0" }
quick-xml.workspace = true
reqwest.workspace = true
ring.workspace = true
rustls.workspace = true
serde.workspace = true
serde_json.workspace = true
Expand All @@ -25,6 +27,7 @@ tokio.workspace = true
tracing.workspace = true
unicode-segmentation.workspace = true
url.workspace = true
zip.workspace = true

[dev-dependencies]
tempfile.workspace = true
Expand Down
132 changes: 127 additions & 5 deletions crates/opsail-read/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# opsail-read

`opsail-read` is the Rust library behind
[`opsail read`](https://github.com/lencx/opsail#read-html). It acquires static HTML
or delegates rendered DOM capture to `opsail-chrome`, extracts the primary
document, sanitizes the result, and returns a versioned `ReadResult` suitable
for agents and other programmatic callers.
[`opsail read`](https://github.com/lencx/opsail#read-html). It acquires static
HTML or delegates rendered DOM capture to `opsail-chrome`, and it sparsely reads
bounded regions from local XLSX workbooks. Both paths return versioned artifacts
suitable for agents and other programmatic callers.

The extraction pipeline is browser-independent. `opsail-read` owns source
validation, non-browser acquisition, extraction, sanitization, and result
Expand All @@ -23,6 +23,20 @@ browser path.
capture one page, and clean up the owned process and profile.
- Resolve relative links and assets against a validated HTTP(S) base URL.
- Extract readable Markdown and sanitized HTML with structured metadata.
- Read local XLSX sheet manifests and repeated `Sheet!A1:D20` ranges without
allocating from declared worksheet dimensions.
- Parse shared and inline strings, stored values, formulas and cached results,
supported date/time number formats, merged ranges, hidden state, and defined
names from OOXML.
- Inventory hyperlinks, filters, tables, conditional formats, validations,
page setup, outlines, drawings, charts, images, comments, controls,
sparklines, themes, and VBA-project presence without executing active content.
- Resolve worksheet picture anchors and media relationships, inventory their
content type/size/SHA-256, and publish bounded data-URI pixels only for
pictures intersecting selected ranges.
- Compare ZIP-part revisions without expanding OOXML, refresh changed worksheet
selections through `WorkbookSession`, and merge generated Markdown blocks
without overwriting agent-authored text.
- Report source, extraction method, quality signals, and warnings through one
stable result model.
- Enforce byte, DOM element, nesting-depth, redirect, and timeout limits.
Expand Down Expand Up @@ -60,6 +74,96 @@ acquisition, leaving `user_agent` as `None` sends `opsail/<version>`; WeChat
article URLs retain their browser-compatible automatic HTTP profile with an
`opsail/<version>` product token. An explicit value always wins.

## Read bounded XLSX ranges

Use `read_artifact` for auto-detection. A local file ending in `.xlsx` returns
`ReadArtifact::Workbook`; other inputs preserve the existing
`ReadArtifact::Document` path:

```rust
use opsail_read::{ReadArtifact, ReadOptions, ReadSource, read_artifact};

async fn inspect(path: std::path::PathBuf) -> Result<(), Box<dyn std::error::Error>> {
let mut options = ReadOptions::default();
options.spreadsheet.ranges = vec![
"Summary!A1:H30".to_owned(),
"'API Sheet'!B6:AY40".to_owned(),
];
let artifact = read_artifact(ReadSource::File(path), &options).await?;
if let ReadArtifact::Workbook(result) = artifact {
println!("{}", serde_json::to_string_pretty(&result)?);
}
Ok(())
}
```

Repeated ranges are planned before extraction. The archive, workbook metadata,
shared strings, and styles are read once, and each selected worksheet XML part
is scanned once. Cells are stored sparsely by reference; a declared dimension
such as `A1:XFD1048576` never causes rectangular allocation.

With no explicit ranges, visible sheets receive a bounded preview controlled by
`preview_rows` and `preview_columns`; hidden sheets remain in the manifest.
`max_cells` bounds published cell records, including blank merge-covered cells,
and marks affected selections as truncated. `max_expanded_bytes` limits cumulative uncompressed OOXML read from
the ZIP package. The normal `max_bytes` limit still applies to the compressed
file itself. Stdout adapters can call `WorkbookReadResult::truncate_published_cells`
and rerender a stable prefix when their serialized transport has a stricter byte
budget than the cell-count limit.

Formula expressions and cached values are reported separately. Opsail does not
recalculate formulas, so callers must not describe the cached value as current.
Legacy `.xls`, encrypted workbooks, macros, chart reconstruction, image OCR,
textbox/VML extraction, conditional-format rendering, and Excel-accurate visual
layout are outside this reader.

The read-side feature levels, tail/cell-data completeness labels, range-overflow
evidence, and human/agent mirror contract are specified in
[`XLSX_COMPATIBILITY.md`](XLSX_COMPATIBILITY.md).

## Refresh a human-owned workbook without losing agent Markdown

`WorkbookSession` is the in-process collaboration API. Keep the XLSX as the
human-owned source and keep agent analysis outside the generated marker blocks:

```rust
use opsail_read::{ReadOptions, WorkbookSession, merge_markdown_mirror};

fn refresh(
path: std::path::PathBuf,
mut mirror: String,
) -> Result<String, Box<dyn std::error::Error>> {
let mut session = WorkbookSession::open(path, ReadOptions::default())?;
if mirror.is_empty() {
mirror = session.result().content.clone();
}

// The agent can append or edit text outside opsail:xlsx-generated blocks.
mirror.push_str("\n\n## Agent analysis\n\nPending human confirmation.\n");
let refresh = session.refresh()?;
Ok(merge_markdown_mirror(&mirror, refresh.result)?)
}
```

An unchanged file uses a metadata fast path and expands zero OOXML bytes. A
worksheet-only change reuses cached shared strings/styles, parses only the
previous selections on changed sheets, and incrementally replaces generated
Markdown/HTML blocks. Workbook, shared-string, style, and theme changes take a
conservative full-refresh path. A full refresh is a correctness fallback, not
an efficiency failure hidden as a partial read.

Run the collaboration benchmark against a workbook tree with:

```sh
cargo +1.97.0 run --release -p opsail-read \
--example xlsx_collaboration_benchmark -- \
--md-rounds 4 --edit-samples 100 /path/to/workbooks
```

The benchmark never changes source files. It edits numeric, non-formula cells
only in temporary copies, compares every incremental selection with a cold
read, and verifies that agent-authored Markdown survives.

## Process caller-captured HTML

Browser hosts should capture the rendered HTML and final page URL themselves,
Expand Down Expand Up @@ -206,7 +310,12 @@ response URL must also match the captured final main document.

## Result contract

Every successful entry point returns `ReadResult`, which contains:
`read` preserves the existing HTML-only `ReadResult`. `read_artifact` returns
the untagged `ReadArtifact` union: existing documents keep their version 1
shape, while workbooks carry `artifactKind: "workbook"` and their own version 1
shape. CLI and Node callers use the latter auto-detected contract.

An HTML `ReadResult` contains:

- `schema_version`: version of the serialized result contract.
- `content` and `content_html`: readable Markdown and sanitized HTML.
Expand All @@ -220,6 +329,19 @@ Every successful entry point returns `ReadResult`, which contains:
Serialized fields use camel case, including `schemaVersion` and `contentHtml`.
Callers should branch on structured fields rather than warning or error text.

A `WorkbookReadResult` contains common `content`, `contentHtml`, `metadata`,
`source`, and `warnings` fields plus:

- `artifactKind: "workbook"`.
- `extraction.method: "ooxml-sparse"`.
- `workbook.sheets`: visibility, declared dimensions, scanned semantic bounds,
merge and hidden-row/column metadata.
- `workbook.definedNames`: scope, reference, and explicit `#REF!` validity.
- `workbook.selections`: requested/resolved bounds, sparse cell records, and
truncation.
- `workbook.statistics`: archive, expanded-byte, scan, style-only, and returned
cell counts.

## Trust boundary

Treat source HTML and all extracted metadata as untrusted input. `opsail-read`
Expand Down
Loading