diff --git a/CHANGELOG.md b/CHANGELOG.md index d0e6c84..acda5e6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,34 @@ All notable changes to FireXCore MailVault are documented in this file. The project follows Semantic Versioning. +## 2.0.6 - 2026-07-18 + +### Added + +- Added resumable navigation-view builds with durable checkpoints and deterministic source fingerprints. +- Added an exact progress bar with source rows, pointer writes, percentage and estimated time remaining. +- Added `mailvault views --restart` for explicitly discarding an incomplete build. +- Added a completed-view snapshot marker so unchanged archives return `UP TO DATE` without rewriting pointers. + +### Fixed + +- Bounded derived view path segments and pointer filenames with collision-resistant hashes for Windows portability. +- Shortened atomic temporary filenames so destination names are not duplicated into temporary paths. +- Built views in a staging tree and published them transactionally, preserving the previous completed views until the replacement is ready. +- Corrected per-view pointer counters so `by-domain`, `by-thread`, `by-year`, `by-mailbox` and `by-label` totals are reported accurately. + +### Reliability + +- A safe interruption checkpoints only fully written source rows; rerunning the same command resumes from the last durable cursor. +- A changed SQLite source snapshot invalidates an incomplete checkpoint and starts a clean replacement build automatically. +- View builds share the archive process lock with sync so derived outputs cannot be published from a concurrently changing archive. + +### Documentation + +- Added a complete operator and engineering guide for Windows-safe, resumable view builds. +- Added real runtime screenshots covering the release quality gate and live view progress. +- Added exact interruption, resume, restart, publication, state-inspection, and troubleshooting procedures. + ## 2.0.5 - 2026-07-14 ### Fixed diff --git a/README.md b/README.md index b99fd36..a486466 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@

راهنمای فارسی · Getting started · + Resumable views · Architecture · Procurement readiness

@@ -215,11 +216,32 @@ See [Configuration](docs/CONFIGURATION.md). | `mailvault stats` | Display message, occurrence, MIME-part, blob, and storage counts. | | `mailvault verify` | Recalculate raw-message and blob hashes. | | `mailvault export` | Regenerate portable JSONL and procurement source manifests. | -| `mailvault views` | Regenerate disposable navigation views by domain, sender, thread, mailbox, year, and label. | +| `mailvault views` | Build resumable navigation views with exact progress, ETA, transactional publication, and automatic checkpoint recovery. | | `mailvault version` | Print the installed distribution version. | See [CLI reference](docs/CLI_REFERENCE.md). +## Windows-safe, resumable navigation views + +MailVault 2.0.6 hardens the derived navigation-view pipeline for large archives: + +1. **Windows-safe paths** — untrusted labels, sender addresses, thread identifiers, and attachment filenames cannot create unbounded view paths. Directory segments and pointer filenames are bounded and receive deterministic SHA-256 suffixes when required. +2. **Durable resume** — completed source rows are checkpointed into a staging build. After `Ctrl+C`, rerunning the same command resumes from the last durable cursor instead of deleting all completed work. +3. **Exact progress and ETA** — the command plans the complete source snapshot, calculates exact source-row and pointer totals, and displays planning, building, resuming, publishing, percentage, and estimated time remaining. + +

+ FireXCore MailVault resumable view build with exact progress and ETA +

+ +```powershell +mailvault views ` + --destination "E:\MailVault-E" +``` + +Use `--restart` only to intentionally discard an incomplete staging build. The previous completed `views/` tree remains available until its replacement is fully written and transactionally published. + +See [Resumable navigation views](docs/RESUMABLE_VIEWS.md) for the lifecycle, state files, path-safety model, recovery behavior, operational checks, and validation commands. + ## Archive layout ```text @@ -302,7 +324,7 @@ Review [Security policy](SECURITY.md) and [Security model](docs/SECURITY_MODEL.m ## Reliability and operational limits -MailVault uses metadata-first discovery, bounded fetch batches, randomized delays, rolling 24-hour bandwidth caps, checkpointed resume, and exponential retry. Provider limits still apply. A sync can be safely stopped and restarted with the same command and destination. +MailVault uses metadata-first discovery, bounded fetch batches, randomized delays, rolling 24-hour bandwidth caps, checkpointed resume, and exponential retry. Provider limits still apply. Sync and navigation-view builds can be safely stopped and restarted with the same command and destination. View builds display exact source-row progress and ETA, write into a resumable staging tree, and publish only after the replacement snapshot is complete. Before transferring or importing an archive, run: @@ -330,6 +352,7 @@ See [Contributing](CONTRIBUTING.md) and [Development](docs/DEVELOPMENT.md). - [Getting started](docs/GETTING_STARTED.md) - [Configuration](docs/CONFIGURATION.md) - [CLI reference](docs/CLI_REFERENCE.md) +- [Resumable navigation views](docs/RESUMABLE_VIEWS.md) - [Provider support](docs/PROVIDERS.md) - [Architecture](docs/ARCHITECTURE.md) - [Archive format](docs/ARCHIVE_FORMAT.md) @@ -341,6 +364,7 @@ See [Contributing](CONTRIBUTING.md) and [Development](docs/DEVELOPMENT.md). - [Repository setup](docs/REPOSITORY_SETUP.md) - [Roadmap](docs/ROADMAP.md) - [References](docs/REFERENCES.md) +- [Release notes: 2.0.6](docs/releases/v2.0.6.md) ## License diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 748a948..7ebf7b3 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -51,7 +51,7 @@ Derived: - navigation views; - reports. -Derived outputs can be removed and regenerated without contacting the mail server. +Derived outputs can be removed and regenerated without contacting the mail server. Navigation views are generated from a fingerprinted SQLite read snapshot, checkpointed by a deterministic row cursor, and published from staging only after completion. ## Identity strategy diff --git a/docs/ARCHIVE_FORMAT.md b/docs/ARCHIVE_FORMAT.md index 6b69617..c88c769 100644 --- a/docs/ARCHIVE_FORMAT.md +++ b/docs/ARCHIVE_FORMAT.md @@ -9,10 +9,14 @@ MailVault/ ├── metadata/messages/.json ├── database/mailvault.sqlite3 ├── manifests/*.jsonl -├── state/* +├── state/ +│ ├── views-rebuild-v1.json +│ ├── views-rebuild-staging-v3/ +│ └── views-previous/ ├── reports/* ├── logs/* -└── views/* +└── views/ + └── _mailvault_views.json ``` ## Raw message objects @@ -55,7 +59,11 @@ JSONL files provide streaming and portable integration surfaces. They are regene ## Navigation views -Views contain pointer JSON, not duplicated attachment bytes. Their paths are sanitized for human navigation and are disposable. +Views contain pointer JSON, not duplicated attachment bytes. Their path segments and pointer names are bounded and collision-resistant for cross-platform portability. Original filenames remain evidence fields inside pointer JSON rather than becoming unbounded storage names. + +Builds use a resumable staging tree, a durable source-row checkpoint, a deterministic source fingerprint, and a completed snapshot marker. The existing completed tree is replaced only after the new snapshot is fully written and ready for publication. An interrupted publication can restore the previous completed tree. Views remain disposable. + +See [Resumable navigation views](RESUMABLE_VIEWS.md). ## Portability diff --git a/docs/CLI_REFERENCE.md b/docs/CLI_REFERENCE.md index 4237340..28e7ead 100644 --- a/docs/CLI_REFERENCE.md +++ b/docs/CLI_REFERENCE.md @@ -98,7 +98,46 @@ Regenerates portable JSONL manifests and `procurement_sources.jsonl` from SQLite ## `mailvault views` -Regenerates disposable pointer views by domain, sender, thread, mailbox, year, and label. +Builds disposable JSON pointer views by domain, sender, thread, mailbox, year, and label. + +```powershell +mailvault views ` + --destination "E:\MailVault-E" +``` + +Options: + +```text +--destination, -d Archive root +--restart Discard an incomplete staging build and start from row zero +``` + +The command performs four phases: + +1. **Planning** scans the protected SQLite snapshot to calculate exact source-row and pointer totals and a deterministic fingerprint. +2. **Building** or **Resuming** writes pointers into `state/views-rebuild-staging-v3/`. +3. **Publishing** atomically replaces the completed `views/` snapshot. +4. **Completed** reports one of `REBUILT`, `RESUMED`, or `UP TO DATE`. + +The progress display includes exact source rows, pointer writes, percentage, and ETA. + +A safe interruption checkpoints fully completed source rows in: + +```text +state/views-rebuild-v1.json +``` + +Rerun the same command to resume. Do not use `--restart` when the goal is to continue the existing staging build. + +The previous completed `views/` tree remains available until the replacement is fully built and published. If SQLite changes after interruption, the stale checkpoint is rejected automatically. If the completed marker and source fingerprint already match, the command returns `UP TO DATE` without rewriting pointer files. + +Completed snapshots contain: + +```text +views/_mailvault_views.json +``` + +See [Resumable navigation views](RESUMABLE_VIEWS.md). ## Module execution diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index abcb022..7be4c0c 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -57,6 +57,22 @@ Provider tests should cover: - provider query translation; - fallback behavior when extensions are absent. + +## View exporter validation + +Run the targeted Windows-path, interruption, resume, progress and snapshot tests: + +```powershell +python -m pytest ` + tests\test_view_exporter.py ` + tests\test_unicode_safety.py ` + -q +``` + +The view exporter tests cover bounded deterministic filenames, short atomic temporary names, durable source-row checkpoints, resume, source-fingerprint invalidation, transactional publication, up-to-date no-op behavior, and exact pointer totals. + +See [Resumable navigation views](RESUMABLE_VIEWS.md). + ## Release validation Before tagging: diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index cf1dda3..7b91214 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -108,6 +108,24 @@ mailvault export --destination E:\MailVault mailvault views --destination E:\MailVault ``` +

+ MailVault live view-build progress +

+ +`mailvault views` first plans an exact source snapshot and then displays determinate source-row progress, exact pointer writes, percentage, and ETA. After `Ctrl+C`, run the identical command to continue from the last durable source-row checkpoint. + +```powershell +mailvault views --destination E:\MailVault +``` + +Use `--restart` only when you explicitly want to discard the incomplete staging build: + +```powershell +mailvault views --destination E:\MailVault --restart +``` + +A successful result reports `REBUILT` or `RESUMED`. A second run against an unchanged archive reports `UP TO DATE` without rewriting pointers. See [Resumable navigation views](RESUMABLE_VIEWS.md). + ## Recommended first-run sequence 1. Run `doctor`. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 91530c6..d64f9d2 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -68,8 +68,27 @@ Rebuild manifests: mailvault export --destination /archive/path ``` -Rebuild navigation views: +Build or resume navigation views: ```bash mailvault views --destination /archive/path ``` + +The progress display reports exact source rows, pointer writes, percentage and ETA. A safe interruption preserves the durable cursor and staging tree; rerun the same command to continue. To intentionally discard an incomplete build: + +```bash +mailvault views --destination /archive/path --restart +``` + +The previous completed `views/` snapshot is retained until the replacement has been fully written and transactionally published. + +Operational state: + +```text +state/views-rebuild-v1.json durable checkpoint +state/views-rebuild-staging-v3/ incomplete replacement tree +state/views-previous/ temporary publication rollback tree +views/_mailvault_views.json completed snapshot marker +``` + +Do not remove the checkpoint or staging tree when the goal is to resume. A source change invalidates stale state automatically. See [Resumable navigation views](RESUMABLE_VIEWS.md) for lifecycle and recovery details. diff --git a/docs/README.fa.md b/docs/README.fa.md index 62d601c..5394ba9 100644 --- a/docs/README.fa.md +++ b/docs/README.fa.md @@ -132,10 +132,47 @@ mailvault sync آرشیو کامل و قابل Resume پیام‌ها mailvault stats نمایش آمار آرشیو mailvault verify بررسی مجدد Hash فایل‌های EML و Blobها mailvault export بازسازی JSONL و Procurement Manifest -mailvault views بازسازی Viewهای Domain، Sender، Thread، Year و Label +mailvault views ساخت Resumeپذیر Viewها با Progress دقیق، ETA و انتشار تراکنشی mailvault version نمایش نسخه نصب‌شده ``` +## Viewهای حرفه‌ای در نسخه 2.0.6 + +نسخه 2.0.6 سه ضعف عملیاتی لایه `views` را برطرف می‌کند: + +1. **مسیرهای امن برای Windows**: نام Label، Sender، Thread و Attachment دیگر نمی‌تواند مسیر نامحدود تولید کند. Segmentها محدود و همراه با Hash پایدار هستند و نام فایل موقت Atomic نیز کوتاه است. +2. **Resume واقعی بعد از توقف**: پس از `Ctrl+C`، Cursor فقط تا آخرین Source Row کاملاً نوشته‌شده جلو می‌رود. اجرای مجدد همان دستور از Checkpoint معتبر ادامه می‌دهد. +3. **Progress و ETA دقیق**: ابتدا تعداد دقیق Source Row و Pointer محاسبه می‌شود؛ سپس مرحله Planning، Building یا Resuming، درصد، تعداد Pointer و زمان تقریبی باقی‌مانده نمایش داده می‌شود. + +

+ نمایش Progress و ETA ساخت Viewهای MailVault +

+ +ساخت یا ادامه Viewها: + +```powershell +mailvault views ` + --destination "E:\MailVault-E" +``` + +توقف با `Ctrl+C` امن است. برای ادامه همان فرمان را دوباره اجرا کن. فقط برای کنارگذاشتن عمدی Build نیمه‌کاره از این گزینه استفاده می‌شود: + +```powershell +mailvault views ` + --destination "E:\MailVault-E" ` + --restart +``` + +وضعیت نهایی یکی از این موارد است: + +```text +REBUILT +RESUMED +UP TO DATE +``` + +تا وقتی Snapshot جدید کامل نشده، View کامل قبلی جایگزین یا حذف نمی‌شود. راهنمای فنی کامل در [Resumable navigation views](RESUMABLE_VIEWS.md) قرار دارد. + ## ارتباط با RMS و Procurement Intelligence فایل زیر برای تمام Bodyها و MIME Partهای قابل استفاده Evidence Record تولید می‌کند: @@ -201,3 +238,4 @@ mailvault verify --destination E:\MailVault - [امنیت](SECURITY_MODEL.md) - [آمادگی Procurement](PROCUREMENT_READINESS.md) - [رفع خطا](TROUBLESHOOTING.md) +- [ساخت Resumeپذیر Viewها](RESUMABLE_VIEWS.md) diff --git a/docs/REPOSITORY_SETUP.md b/docs/REPOSITORY_SETUP.md index a018e8c..c0ef580 100644 --- a/docs/REPOSITORY_SETUP.md +++ b/docs/REPOSITORY_SETUP.md @@ -50,7 +50,7 @@ Enable: ## Releases -Create Git tags in the form `v2.0.5`. The release workflow builds wheel and source distribution, validates metadata, writes SHA-256 checksums and attaches artifacts to the GitHub release. +Create Git tags in the form `v2.0.6`. The release workflow builds wheel and source distribution, validates metadata, writes SHA-256 checksums and attaches artifacts to the GitHub release. PyPI publishing uses a separate trusted-publishing workflow. Configure the GitHub `pypi` environment and PyPI trusted publisher, then set the repository variable `PYPI_PUBLISH_ENABLED=true`. Manual dispatch remains available for controlled validation. diff --git a/docs/RESUMABLE_VIEWS.md b/docs/RESUMABLE_VIEWS.md new file mode 100644 index 0000000..62dbcdf --- /dev/null +++ b/docs/RESUMABLE_VIEWS.md @@ -0,0 +1,330 @@ +# Resumable navigation views + +FireXCore MailVault navigation views are disposable JSON pointer trees derived from the canonical SQLite index and immutable object store. They make large archives easier to browse by sender domain, thread, year, mailbox, and Gmail label without duplicating raw EML or attachment bytes. + +Version 2.0.6 replaces the original one-shot view exporter with a Windows-safe, resumable, observable build pipeline. + +

+ MailVault 2.0.6 view build running with exact progress and ETA +

+ +## What changed in 2.0.6 + +| Change | Previous behavior | Version 2.0.6 behavior | +|---|---|---| +| Path safety | Long attachment names could produce paths that exceeded Windows limits. | Directory segments and pointer filenames are bounded and collision-resistant; atomic temporary filenames use a short constant prefix. | +| Interruption handling | `Ctrl+C` discarded all partial work on the next run. | Fully completed source rows are checkpointed and the same command resumes from the durable cursor. | +| Progress visibility | The command displayed output only after the entire build completed. | Planning, building, resuming, publishing, exact row counts, pointer counts, percentage, and ETA are displayed live. | + +These changes affect only the derived `views/` layer. Raw EML objects, content-addressed blobs, SQLite records, manifests, reports, and evidence files are not rewritten by `mailvault views`. + +## View model + +Each pointer is a small JSON document that references canonical archive objects. A pointer may include: + +- canonical message archive ID; +- provider thread identity; +- subject; +- raw EML path; +- MIME part ID and path; +- original attachment filename; +- SHA-256 digest; +- blob path; +- detected MIME type and size. + +The original attachment filename remains inside pointer JSON for evidence and inspection. It is not trusted as an unbounded filesystem filename. + +The published tree contains: + +```text +views/ +├── _mailvault_views.json +├── by-domain////.json +├── by-thread//.json +├── by-year//.json +├── by-mailbox//.json +└── by-label/