Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,34 @@ All notable changes to FireXCore MailVault are documented in this file.

The project follows Semantic Versioning.

## 2.0.6 - 2026-07-18

### Added

- Added resumable navigation-view builds with durable checkpoints and deterministic source fingerprints.
- Added an exact progress bar with source rows, pointer writes, percentage and estimated time remaining.
- Added `mailvault views --restart` for explicitly discarding an incomplete build.
- Added a completed-view snapshot marker so unchanged archives return `UP TO DATE` without rewriting pointers.

### Fixed

- Bounded derived view path segments and pointer filenames with collision-resistant hashes for Windows portability.
- Shortened atomic temporary filenames so destination names are not duplicated into temporary paths.
- Built views in a staging tree and published them transactionally, preserving the previous completed views until the replacement is ready.
- Corrected per-view pointer counters so `by-domain`, `by-thread`, `by-year`, `by-mailbox` and `by-label` totals are reported accurately.

### Reliability

- A safe interruption checkpoints only fully written source rows; rerunning the same command resumes from the last durable cursor.
- A changed SQLite source snapshot invalidates an incomplete checkpoint and starts a clean replacement build automatically.
- View builds share the archive process lock with sync so derived outputs cannot be published from a concurrently changing archive.

### Documentation

- Added a complete operator and engineering guide for Windows-safe, resumable view builds.
- Added real runtime screenshots covering the release quality gate and live view progress.
- Added exact interruption, resume, restart, publication, state-inspection, and troubleshooting procedures.

## 2.0.5 - 2026-07-14

### Fixed
Expand Down
28 changes: 26 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
<p align="center">
<a href="docs/README.fa.md">راهنمای فارسی</a> ·
<a href="docs/GETTING_STARTED.md">Getting started</a> ·
<a href="docs/RESUMABLE_VIEWS.md">Resumable views</a> ·
<a href="docs/ARCHITECTURE.md">Architecture</a> ·
<a href="docs/PROCUREMENT_READINESS.md">Procurement readiness</a>
</p>
Expand Down Expand Up @@ -215,11 +216,32 @@ See [Configuration](docs/CONFIGURATION.md).
| `mailvault stats` | Display message, occurrence, MIME-part, blob, and storage counts. |
| `mailvault verify` | Recalculate raw-message and blob hashes. |
| `mailvault export` | Regenerate portable JSONL and procurement source manifests. |
| `mailvault views` | Regenerate disposable navigation views by domain, sender, thread, mailbox, year, and label. |
| `mailvault views` | Build resumable navigation views with exact progress, ETA, transactional publication, and automatic checkpoint recovery. |
| `mailvault version` | Print the installed distribution version. |

See [CLI reference](docs/CLI_REFERENCE.md).

## Windows-safe, resumable navigation views

MailVault 2.0.6 hardens the derived navigation-view pipeline for large archives:

1. **Windows-safe paths** — untrusted labels, sender addresses, thread identifiers, and attachment filenames cannot create unbounded view paths. Directory segments and pointer filenames are bounded and receive deterministic SHA-256 suffixes when required.
2. **Durable resume** — completed source rows are checkpointed into a staging build. After `Ctrl+C`, rerunning the same command resumes from the last durable cursor instead of deleting all completed work.
3. **Exact progress and ETA** — the command plans the complete source snapshot, calculates exact source-row and pointer totals, and displays planning, building, resuming, publishing, percentage, and estimated time remaining.

<p align="center">
<img src="docs/assets/views-resume-progress.png" alt="FireXCore MailVault resumable view build with exact progress and ETA" width="100%">
</p>

```powershell
mailvault views `
--destination "E:\MailVault-E"
```

Use `--restart` only to intentionally discard an incomplete staging build. The previous completed `views/` tree remains available until its replacement is fully written and transactionally published.

See [Resumable navigation views](docs/RESUMABLE_VIEWS.md) for the lifecycle, state files, path-safety model, recovery behavior, operational checks, and validation commands.

## Archive layout

```text
Expand Down Expand Up @@ -302,7 +324,7 @@ Review [Security policy](SECURITY.md) and [Security model](docs/SECURITY_MODEL.m

## Reliability and operational limits

MailVault uses metadata-first discovery, bounded fetch batches, randomized delays, rolling 24-hour bandwidth caps, checkpointed resume, and exponential retry. Provider limits still apply. A sync can be safely stopped and restarted with the same command and destination.
MailVault uses metadata-first discovery, bounded fetch batches, randomized delays, rolling 24-hour bandwidth caps, checkpointed resume, and exponential retry. Provider limits still apply. Sync and navigation-view builds can be safely stopped and restarted with the same command and destination. View builds display exact source-row progress and ETA, write into a resumable staging tree, and publish only after the replacement snapshot is complete.

Before transferring or importing an archive, run:

Expand Down Expand Up @@ -330,6 +352,7 @@ See [Contributing](CONTRIBUTING.md) and [Development](docs/DEVELOPMENT.md).
- [Getting started](docs/GETTING_STARTED.md)
- [Configuration](docs/CONFIGURATION.md)
- [CLI reference](docs/CLI_REFERENCE.md)
- [Resumable navigation views](docs/RESUMABLE_VIEWS.md)
- [Provider support](docs/PROVIDERS.md)
- [Architecture](docs/ARCHITECTURE.md)
- [Archive format](docs/ARCHIVE_FORMAT.md)
Expand All @@ -341,6 +364,7 @@ See [Contributing](CONTRIBUTING.md) and [Development](docs/DEVELOPMENT.md).
- [Repository setup](docs/REPOSITORY_SETUP.md)
- [Roadmap](docs/ROADMAP.md)
- [References](docs/REFERENCES.md)
- [Release notes: 2.0.6](docs/releases/v2.0.6.md)

## License

Expand Down
2 changes: 1 addition & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Derived:
- navigation views;
- reports.

Derived outputs can be removed and regenerated without contacting the mail server.
Derived outputs can be removed and regenerated without contacting the mail server. Navigation views are generated from a fingerprinted SQLite read snapshot, checkpointed by a deterministic row cursor, and published from staging only after completion.

## Identity strategy

Expand Down
14 changes: 11 additions & 3 deletions docs/ARCHIVE_FORMAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,14 @@ MailVault/
├── metadata/messages/<archive-id>.json
├── database/mailvault.sqlite3
├── manifests/*.jsonl
├── state/*
├── state/
│ ├── views-rebuild-v1.json
│ ├── views-rebuild-staging-v3/
│ └── views-previous/
├── reports/*
├── logs/*
└── views/*
└── views/
└── _mailvault_views.json
```

## Raw message objects
Expand Down Expand Up @@ -55,7 +59,11 @@ JSONL files provide streaming and portable integration surfaces. They are regene

## Navigation views

Views contain pointer JSON, not duplicated attachment bytes. Their paths are sanitized for human navigation and are disposable.
Views contain pointer JSON, not duplicated attachment bytes. Their path segments and pointer names are bounded and collision-resistant for cross-platform portability. Original filenames remain evidence fields inside pointer JSON rather than becoming unbounded storage names.

Builds use a resumable staging tree, a durable source-row checkpoint, a deterministic source fingerprint, and a completed snapshot marker. The existing completed tree is replaced only after the new snapshot is fully written and ready for publication. An interrupted publication can restore the previous completed tree. Views remain disposable.

See [Resumable navigation views](RESUMABLE_VIEWS.md).

## Portability

Expand Down
41 changes: 40 additions & 1 deletion docs/CLI_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,46 @@ Regenerates portable JSONL manifests and `procurement_sources.jsonl` from SQLite

## `mailvault views`

Regenerates disposable pointer views by domain, sender, thread, mailbox, year, and label.
Builds disposable JSON pointer views by domain, sender, thread, mailbox, year, and label.

```powershell
mailvault views `
--destination "E:\MailVault-E"
```

Options:

```text
--destination, -d Archive root
--restart Discard an incomplete staging build and start from row zero
```

The command performs four phases:

1. **Planning** scans the protected SQLite snapshot to calculate exact source-row and pointer totals and a deterministic fingerprint.
2. **Building** or **Resuming** writes pointers into `state/views-rebuild-staging-v3/`.
3. **Publishing** atomically replaces the completed `views/` snapshot.
4. **Completed** reports one of `REBUILT`, `RESUMED`, or `UP TO DATE`.

The progress display includes exact source rows, pointer writes, percentage, and ETA.

A safe interruption checkpoints fully completed source rows in:

```text
state/views-rebuild-v1.json
```

Rerun the same command to resume. Do not use `--restart` when the goal is to continue the existing staging build.

The previous completed `views/` tree remains available until the replacement is fully built and published. If SQLite changes after interruption, the stale checkpoint is rejected automatically. If the completed marker and source fingerprint already match, the command returns `UP TO DATE` without rewriting pointer files.

Completed snapshots contain:

```text
views/_mailvault_views.json
```

See [Resumable navigation views](RESUMABLE_VIEWS.md).

## Module execution

Expand Down
16 changes: 16 additions & 0 deletions docs/DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,22 @@ Provider tests should cover:
- provider query translation;
- fallback behavior when extensions are absent.


## View exporter validation

Run the targeted Windows-path, interruption, resume, progress and snapshot tests:

```powershell
python -m pytest `
tests\test_view_exporter.py `
tests\test_unicode_safety.py `
-q
```

The view exporter tests cover bounded deterministic filenames, short atomic temporary names, durable source-row checkpoints, resume, source-fingerprint invalidation, transactional publication, up-to-date no-op behavior, and exact pointer totals.

See [Resumable navigation views](RESUMABLE_VIEWS.md).

## Release validation

Before tagging:
Expand Down
18 changes: 18 additions & 0 deletions docs/GETTING_STARTED.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,24 @@ mailvault export --destination E:\MailVault
mailvault views --destination E:\MailVault
```

<p align="center">
<img src="assets/views-progress-terminal.png" alt="MailVault live view-build progress" width="100%">
</p>

`mailvault views` first plans an exact source snapshot and then displays determinate source-row progress, exact pointer writes, percentage, and ETA. After `Ctrl+C`, run the identical command to continue from the last durable source-row checkpoint.

```powershell
mailvault views --destination E:\MailVault
```

Use `--restart` only when you explicitly want to discard the incomplete staging build:

```powershell
mailvault views --destination E:\MailVault --restart
```

A successful result reports `REBUILT` or `RESUMED`. A second run against an unchanged archive reports `UP TO DATE` without rewriting pointers. See [Resumable navigation views](RESUMABLE_VIEWS.md).

## Recommended first-run sequence

1. Run `doctor`.
Expand Down
21 changes: 20 additions & 1 deletion docs/OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,27 @@ Rebuild manifests:
mailvault export --destination /archive/path
```

Rebuild navigation views:
Build or resume navigation views:

```bash
mailvault views --destination /archive/path
```

The progress display reports exact source rows, pointer writes, percentage and ETA. A safe interruption preserves the durable cursor and staging tree; rerun the same command to continue. To intentionally discard an incomplete build:

```bash
mailvault views --destination /archive/path --restart
```

The previous completed `views/` snapshot is retained until the replacement has been fully written and transactionally published.

Operational state:

```text
state/views-rebuild-v1.json durable checkpoint
state/views-rebuild-staging-v3/ incomplete replacement tree
state/views-previous/ temporary publication rollback tree
views/_mailvault_views.json completed snapshot marker
```

Do not remove the checkpoint or staging tree when the goal is to resume. A source change invalidates stale state automatically. See [Resumable navigation views](RESUMABLE_VIEWS.md) for lifecycle and recovery details.
40 changes: 39 additions & 1 deletion docs/README.fa.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,47 @@ mailvault sync آرشیو کامل و قابل Resume پیام‌ها
mailvault stats نمایش آمار آرشیو
mailvault verify بررسی مجدد Hash فایل‌های EML و Blobها
mailvault export بازسازی JSONL و Procurement Manifest
mailvault views بازسازی Viewهای Domain، Sender، Thread، Year و Label
mailvault views ساخت Resumeپذیر Viewها با Progress دقیق، ETA و انتشار تراکنشی
mailvault version نمایش نسخه نصب‌شده
```

## Viewهای حرفه‌ای در نسخه 2.0.6

نسخه 2.0.6 سه ضعف عملیاتی لایه `views` را برطرف می‌کند:

1. **مسیرهای امن برای Windows**: نام Label، Sender، Thread و Attachment دیگر نمی‌تواند مسیر نامحدود تولید کند. Segmentها محدود و همراه با Hash پایدار هستند و نام فایل موقت Atomic نیز کوتاه است.
2. **Resume واقعی بعد از توقف**: پس از `Ctrl+C`، Cursor فقط تا آخرین Source Row کاملاً نوشته‌شده جلو می‌رود. اجرای مجدد همان دستور از Checkpoint معتبر ادامه می‌دهد.
3. **Progress و ETA دقیق**: ابتدا تعداد دقیق Source Row و Pointer محاسبه می‌شود؛ سپس مرحله Planning، Building یا Resuming، درصد، تعداد Pointer و زمان تقریبی باقی‌مانده نمایش داده می‌شود.

<p align="center">
<img src="assets/views-progress-terminal.png" alt="نمایش Progress و ETA ساخت Viewهای MailVault" width="100%">
</p>

ساخت یا ادامه Viewها:

```powershell
mailvault views `
--destination "E:\MailVault-E"
```

توقف با `Ctrl+C` امن است. برای ادامه همان فرمان را دوباره اجرا کن. فقط برای کنارگذاشتن عمدی Build نیمه‌کاره از این گزینه استفاده می‌شود:

```powershell
mailvault views `
--destination "E:\MailVault-E" `
--restart
```

وضعیت نهایی یکی از این موارد است:

```text
REBUILT
RESUMED
UP TO DATE
```

تا وقتی Snapshot جدید کامل نشده، View کامل قبلی جایگزین یا حذف نمی‌شود. راهنمای فنی کامل در [Resumable navigation views](RESUMABLE_VIEWS.md) قرار دارد.

## ارتباط با RMS و Procurement Intelligence

فایل زیر برای تمام Bodyها و MIME Partهای قابل استفاده Evidence Record تولید می‌کند:
Expand Down Expand Up @@ -201,3 +238,4 @@ mailvault verify --destination E:\MailVault
- [امنیت](SECURITY_MODEL.md)
- [آمادگی Procurement](PROCUREMENT_READINESS.md)
- [رفع خطا](TROUBLESHOOTING.md)
- [ساخت Resumeپذیر Viewها](RESUMABLE_VIEWS.md)
2 changes: 1 addition & 1 deletion docs/REPOSITORY_SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Enable:

## Releases

Create Git tags in the form `v2.0.5`. The release workflow builds wheel and source distribution, validates metadata, writes SHA-256 checksums and attaches artifacts to the GitHub release.
Create Git tags in the form `v2.0.6`. The release workflow builds wheel and source distribution, validates metadata, writes SHA-256 checksums and attaches artifacts to the GitHub release.

PyPI publishing uses a separate trusted-publishing workflow. Configure the GitHub `pypi` environment and PyPI trusted publisher, then set the repository variable `PYPI_PUBLISH_ENABLED=true`. Manual dispatch remains available for controlled validation.

Expand Down
Loading