fix: guarantee complete Gmail scope and label coverage

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -12,7 +12,7 @@ body:
     id: version
     attributes:
       label: MailVault version
-      placeholder: 2.0.4
+      placeholder: 2.0.5
     validations:
       required: true
 

CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to FireXCore MailVault are documented in this file.
 
 The project follows Semantic Versioning.
 
+## 2.0.5 - 2026-07-14
+
+### Fixed
+
+- Corrected Gmail full-scope discovery to use the standard IMAP `SEARCH ALL` criterion.
+- Removed the invalid `X-GM-RAW "all"` behavior, which interpreted `all` as a Gmail text search and could silently omit older messages.
+- Versioned scan selection keys so existing archives perform a safe full metadata rescan after upgrading instead of reusing the affected checkpoint.
+
+### Added
+
+- Added Gmail message-ID fetch support for remote label reconciliation.
+- Added the `mailvault audit-labels` command to compare every IMAP-visible Gmail label with locally archived raw EML identities.
+- Full-scope Gmail sync now runs the same remote label audit before reporting `complete`; missing remote messages produce `incomplete` and exit code 2.
+
 ## 2.0.4 - 2026-07-13
 
 ### Fixed

CITATION.cff

@@ -8,5 +8,5 @@ authors:
 repository-code: "https://github.com/FireXCore/mailvault"
 url: "https://github.com/FireXCore/mailvault"
 license: "Apache-2.0"
-version: "2.0.4"
+version: "2.0.5"
 date-released: "2026-07-13"

README.md

@@ -149,6 +149,17 @@ mailvault sync `
 
 The App Password is requested through hidden terminal input. It is not written to configuration, SQLite, JSON, manifests, reports, or logs.
 
+After a full Gmail sync, verify remote label coverage before treating the archive as final:
+
+```powershell
+mailvault audit-labels `
+  --account user@gmail.com `
+  --host imap.gmail.com `
+  --destination E:\MailVault
+```
+
+Full-scope Gmail sync performs this reconciliation automatically before it can report `complete`. The standalone command can be rerun at any time and writes a JSON report under `reports/`.
+
 <p align="center">
   <img src="docs/assets/doctor-output.svg" alt="MailVault doctor output" width="100%">
 </p>
@@ -200,6 +211,7 @@ See [Configuration](docs/CONFIGURATION.md).
 |---|---|
 | `mailvault doctor` | Validate TLS, authentication, server capabilities, provider profile, and mailbox discovery. |
 | `mailvault sync` | Discover metadata and archive complete raw messages with resumable state. |
+| `mailvault audit-labels` | Compare every IMAP-visible Gmail label with locally archived raw EML identities. |
 | `mailvault stats` | Display message, occurrence, MIME-part, blob, and storage counts. |
 | `mailvault verify` | Recalculate raw-message and blob hashes. |
 | `mailvault export` | Regenerate portable JSONL and procurement source manifests. |

docs/CLI_REFERENCE.md

@@ -39,7 +39,7 @@ Important options:
 --query             Provider query used with scope=query
 --include-spam      Include Spam/Junk archive roots
 --include-trash     Include Trash archive roots
---mailbox           Repeatable generic-IMAP mailbox pattern
+--mailbox           Repeatable mailbox pattern; Gmail labels are supported
 --soft-cap          Rolling 24-hour soft download cap
 --hard-cap          Rolling 24-hour hard download cap
 --log-level         DEBUG | INFO | WARNING | ERROR
@@ -59,6 +59,23 @@ Incorrect:
 --auth <actual-secret>
 ```
 
+
+## `mailvault audit-labels`
+
+Performs a read-only Gmail reconciliation. It enumerates every IMAP-visible mailbox or label, retrieves `X-GM-MSGID` values, and compares them with locally archived raw EML identities.
+
+```text
+--account, -a       Gmail account present in the archive
+--destination, -d   Archive root
+--host              Gmail IMAP hostname; default imap.gmail.com
+--port              IMAP port; default 993
+--auth              app-password | password
+--tls-mode          implicit | starttls
+--timeout           Socket timeout in seconds
+```
+
+The command writes a timestamped JSON report under `reports/` and exits with code `2` when any remote Gmail message lacks raw EML. Full-scope Gmail sync performs the same audit automatically before reporting `complete`. The audit never downloads raw message bodies and does not mutate the mailbox.
+
 ## `mailvault stats`
 
 ```text

docs/GETTING_STARTED.md

@@ -82,6 +82,19 @@ mailvault sync `
 
 The first phase discovers metadata. The second phase fetches pending raw messages. Stopping the process does not invalidate the archive; use the same destination and command to resume.
 
+## Audit Gmail label coverage
+
+For Gmail, full-scope sync performs remote label reconciliation before reporting `complete`. You can also rerun the audit explicitly before final verification:
+
+```powershell
+mailvault audit-labels `
+  --account user@gmail.com `
+  --host imap.gmail.com `
+  --destination E:\MailVault
+```
+
+A passing audit proves that every Gmail message visible through those labels has a corresponding archived raw EML identity.
+
 ## Verify the archive
 
 ```powershell
@@ -100,6 +113,7 @@ mailvault views --destination E:\MailVault
 1. Run `doctor`.
 2. Run a narrow date or query scope against a new destination when validating a new provider.
 3. Inspect `reports`, `logs`, `metadata/messages`, and `manifests`.
-4. Run `verify`.
-5. Start the full `scope=all` archive.
-6. Preserve the destination and resume rather than restarting from an empty directory.
+4. Start the full `scope=all` archive.
+5. For Gmail, run `audit-labels` and require a passing result.
+6. Run `verify`.
+7. Preserve the destination and resume rather than restarting from an empty directory.

docs/REPOSITORY_SETUP.md

@@ -50,7 +50,7 @@ Enable:
 
 ## Releases
 
-Create Git tags in the form `v2.0.4`. The release workflow builds wheel and source distribution, validates metadata, writes SHA-256 checksums and attaches artifacts to the GitHub release.
+Create Git tags in the form `v2.0.5`. The release workflow builds wheel and source distribution, validates metadata, writes SHA-256 checksums and attaches artifacts to the GitHub release.
 
 PyPI publishing uses a separate trusted-publishing workflow. Configure the GitHub `pypi` environment and PyPI trusted publisher, then set the repository variable `PYPI_PUBLISH_ENABLED=true`. Manual dispatch remains available for controlled validation.
 

docs/TROUBLESHOOTING.md

@@ -71,3 +71,18 @@ Do not delete it. Run the same version and command against the same destination.
 ## Integrity verification fails
 
 Do not continue with downstream ingestion. Check missing paths, storage corruption, interrupted external copy operations, antivirus quarantine and filesystem errors. Restore canonical objects from backup, then rerun full verification.
+
+## Gmail archive created with version 2.0.4 or earlier is incomplete
+
+Versions through 2.0.4 used `X-GM-RAW "all"` for `scope=all`. Gmail interprets X-GM-RAW arguments using Gmail web-search syntax, so the bare word `all` is a text query rather than a universal match. An affected run can therefore report `complete` after archiving only messages matching that search term.
+
+Upgrade to 2.0.5 or newer and rerun the same full-scope command against the existing destination. Versioned scan keys force a safe metadata rescan without discarding already archived objects. Then run:
+
+```powershell
+mailvault audit-labels `
+  --account user@gmail.com `
+  --host imap.gmail.com `
+  --destination E:\MailVault
+```
+
+Do not finalize exports or backups until label coverage passes and `mailvault verify --sample 1` succeeds.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "firexcore-mailvault"
-version = "2.0.4"
+version = "2.0.5"
 description = "Provider-neutral, read-only, evidence-preserving email archival for analytics, eDiscovery, and procurement intelligence."
 readme = "README.md"
 requires-python = ">=3.12"

src/firexcore_mailvault/cli.py

@@ -15,6 +15,7 @@
 from firexcore_mailvault.config import MailVaultConfig, config_from_toml, parse_bytes
 from firexcore_mailvault.errors import MailVaultError
 from firexcore_mailvault.exporter import export_jsonl
+from firexcore_mailvault.gmail_audit import audit_gmail_labels, write_gmail_label_audit
 from firexcore_mailvault.lock import RunLock
 from firexcore_mailvault.logging_setup import configure_logging
 from firexcore_mailvault.models import ArchiveScope, AuthKind, ProviderKind, SyncSummary, TlsMode
@@ -197,6 +198,8 @@ def on_progress(event: str, payload: dict[str, object]) -> None:
                         paths.manifests / "procurement_sources.jsonl"
                     )
             _print_sync_summary(summary)
+            if summary.status == "incomplete":
+                raise typer.Exit(2)
         except KeyboardInterrupt as exc:
             console.print(
                 "[yellow]Sync interrupted safely. Run the same command to resume.[/yellow]"
@@ -207,6 +210,80 @@ def on_progress(event: str, payload: dict[str, object]) -> None:
             raise typer.Exit(1) from exc
 
 
+@app.command("audit-labels")
+def audit_labels_command(
+    account: Annotated[str, typer.Option("--account", "-a", prompt=True)],
+    destination: Annotated[Path, typer.Option("--destination", "-d")],
+    host: Annotated[str, typer.Option("--host")] = "imap.gmail.com",
+    port: Annotated[int, typer.Option("--port")] = 993,
+    auth: Annotated[AuthKind, typer.Option("--auth")] = AuthKind.APP_PASSWORD,
+    tls_mode: Annotated[TlsMode, typer.Option("--tls-mode")] = TlsMode.IMPLICIT,
+    timeout: Annotated[int, typer.Option("--timeout")] = 90,
+) -> None:
+    """Compare every IMAP-visible Gmail label with locally archived raw EML identities."""
+    paths = build_archive_paths(destination)
+    configure_logging(paths.logs / "mailvault.jsonl", "INFO")
+    secret = _secret(auth)
+
+    try:
+        with (
+            ArchiveRepository(paths.database) as repository,
+            ImapGateway(
+                account,
+                secret,
+                host=host,
+                port=port,
+                tls_mode=tls_mode,
+                timeout_seconds=timeout,
+                client_contact="https://github.com/FireXCore/mailvault/issues",
+            ) as gateway,
+        ):
+            profile = resolve_provider(ProviderKind.GMAIL, gateway.capabilities)
+            profile.validate_capabilities(gateway.capabilities)
+            account_id = repository.find_account_id(account)
+            if account_id is None:
+                raise typer.BadParameter("Account does not exist in this archive.")
+
+            report = audit_gmail_labels(
+                gateway,
+                repository,
+                account_id=account_id,
+                account=account,
+            )
+            report_path = write_gmail_label_audit(report, paths.reports)
+
+        table = Table(title="Gmail Label Coverage Audit")
+        table.add_column("Mailbox")
+        table.add_column("Remote", justify="right")
+        table.add_column("Raw", justify="right")
+        table.add_column("Missing", justify="right")
+        table.add_column("Result")
+        for item in report.labels:
+            table.add_row(
+                item.mailbox,
+                f"{item.remote_messages:,}",
+                f"{item.archived_raw_messages:,}",
+                f"{item.missing_raw_messages:,}",
+                "PASS" if item.passed else "FAIL",
+            )
+        console.print(table)
+        console.print(f"Report: {report_path}")
+
+        if not report.passed:
+            console.print(
+                f"[red]Coverage failed:[/red] {report.missing_raw_messages:,} "
+                "unique Gmail messages do not have archived raw EML."
+            )
+            raise typer.Exit(2)
+
+        console.print("[green]Coverage PASS:[/green] every IMAP-visible label is archived.")
+    except typer.Exit:
+        raise
+    except Exception as exc:
+        console.print(f"[red]Label audit failed:[/red] {sanitize_text(str(exc))}")
+        raise typer.Exit(1) from exc
+
+
 @app.command("stats")
 def stats_command(
     destination: Annotated[Path, typer.Option("--destination", "-d")],

src/firexcore_mailvault/gmail_audit.py

@@ -0,0 +1,112 @@
+from __future__ import annotations
+
+from dataclasses import asdict, dataclass
+from datetime import UTC, datetime
+from pathlib import Path
+
+from firexcore_mailvault.atomic import atomic_write_json
+from firexcore_mailvault.models import MailboxInfo
+from firexcore_mailvault.protocols.imap import ImapGatewayProtocol
+from firexcore_mailvault.repository import ArchiveRepository
+
+
+@dataclass(frozen=True, slots=True)
+class GmailLabelCoverage:
+    mailbox: str
+    flags: tuple[str, ...]
+    remote_messages: int
+    archived_raw_messages: int
+    missing_raw_messages: int
+    missing_gmail_message_ids: tuple[str, ...]
+
+    @property
+    def passed(self) -> bool:
+        return self.missing_raw_messages == 0
+
+
+@dataclass(frozen=True, slots=True)
+class GmailLabelAuditReport:
+    account: str
+    generated_at: str
+    local_raw_message_ids: int
+    labels: tuple[GmailLabelCoverage, ...]
+
+    @property
+    def passed(self) -> bool:
+        return all(item.passed for item in self.labels)
+
+    @property
+    def missing_raw_messages(self) -> int:
+        return len(
+            {gmail_id for label in self.labels for gmail_id in label.missing_gmail_message_ids}
+        )
+
+    def as_dict(self) -> dict[str, object]:
+        return {
+            "account": self.account,
+            "generated_at": self.generated_at,
+            "local_raw_message_ids": self.local_raw_message_ids,
+            "passed": self.passed,
+            "missing_raw_messages": self.missing_raw_messages,
+            "labels": [
+                {
+                    **asdict(label),
+                    "passed": label.passed,
+                }
+                for label in self.labels
+            ],
+        }
+
+
+def audit_gmail_labels(
+    gateway: ImapGatewayProtocol,
+    repository: ArchiveRepository,
+    *,
+    account_id: int,
+    account: str,
+    batch_size: int = 500,
+) -> GmailLabelAuditReport:
+    local_raw_ids = repository.gmail_raw_message_ids(account_id)
+    coverages: list[GmailLabelCoverage] = []
+
+    for mailbox in _selectable_mailboxes(gateway.list_mailboxes()):
+        gateway.select_readonly(mailbox.name)
+        uids = sorted(set(gateway.search_uids(("imap", "ALL"))))
+        remote_ids: set[str] = set()
+
+        for index in range(0, len(uids), batch_size):
+            batch = uids[index : index + batch_size]
+            remote_ids.update(gateway.fetch_gmail_message_ids(batch).values())
+
+        missing = tuple(sorted(remote_ids - local_raw_ids))
+        coverages.append(
+            GmailLabelCoverage(
+                mailbox=mailbox.name,
+                flags=mailbox.flags,
+                remote_messages=len(remote_ids),
+                archived_raw_messages=len(remote_ids & local_raw_ids),
+                missing_raw_messages=len(missing),
+                missing_gmail_message_ids=missing,
+            )
+        )
+
+    return GmailLabelAuditReport(
+        account=account,
+        generated_at=datetime.now(UTC).isoformat(),
+        local_raw_message_ids=len(local_raw_ids),
+        labels=tuple(coverages),
+    )
+
+
+def write_gmail_label_audit(report: GmailLabelAuditReport, reports_dir: Path) -> Path:
+    timestamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
+    path = reports_dir / f"gmail-label-audit-{timestamp}.json"
+    atomic_write_json(path, report.as_dict())
+    return path
+
+
+def _selectable_mailboxes(mailboxes: list[MailboxInfo]) -> list[MailboxInfo]:
+    return sorted(
+        (mailbox for mailbox in mailboxes if mailbox.selectable),
+        key=lambda item: item.name.casefold(),
+    )