feat(crypto): optional transparent at-rest AES-256-GCM encryption - #84
Closed
Xyvran wants to merge 28 commits into
Closed
feat(crypto): optional transparent at-rest AES-256-GCM encryption#84Xyvran wants to merge 28 commits into
Xyvran wants to merge 28 commits into
Conversation
Add an optional, transparent encryption layer that sits between the file content API and the Telegram storage backend. When enabled via `tgfs.encryption.enabled`, every byte uploaded to Telegram is encrypted client-side; the channel and the metadata store only ever see ciphertext. Design: * AES-256-GCM in 64 KiB chunks, each with its own nonce + auth tag. * Per-file key via HKDF-SHA256 from master key + 32-byte random salt. * Master key derived from passphrase via Argon2id at startup. * 60-byte self-describing file header (with HMAC) prepended inline to the first Telegram part, so a file can be decrypted from the channel alone even if TGFS metadata is lost. * Decorator pattern around IFileContentRepository -- the rest of TGFS is unchanged. WebDAV range requests still work via per-chunk random access. Configuration: * New `tgfs.encryption` block in config.yaml with passphrase / passphrase_env / passphrase_file options (mutually exclusive). * `master_salt_file` holds the 16-byte Argon2 salt; created on first run. Test coverage: 67 new tests (49 unit + 18 end-to-end) including round-trip across multiple chunk sizes, chunk-boundary range requests, tampering detection, wrong master key detection, and empty-file handling. All 487 existing tests continue to pass.
When the encryption decorator is installed, reads now sniff the first four bytes of each file: a TGFS magic match routes through the existing decrypt path, anything else is streamed back from the inner repo unchanged. This lets a deployment turn encryption on without losing access to files that were uploaded before the switch. Writes are unaffected -- save() and update() keep encrypting whenever the wrapper is active, so overwriting a legacy plaintext file produces ciphertext as expected. Detection failures are loud, not silent: a file whose first four bytes are "TGFS" but whose header is truncated, malformed, or fails MAC verification still raises InvalidHeaderError. Silently falling back to plaintext there would mask a wrong master key. The header cache now stores a None sentinel for plaintext-detected files, so the magic-byte probe runs at most once per file id. 13 new tests cover the passthrough path (full reads, range reads, files shorter than the header / magic), the overwrite-plaintext-with-cipher case, two loud-failure cases for ambiguous TGFS-prefixed plaintext, and the detection cache. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The encryption decorator was internally treating `end` as a Python-slice exclusive bound, while the rest of `IFileContentRepository` (and the underlying Telegram `download_file` API: `bytes_to_read = end - begin + 1`) uses HTTP-Range inclusive ends. The off-by-one made the wrapper pull one extra ciphertext byte at chunk boundaries, which threw AES-GCM tag verification on the following chunk and aborted the stream mid-response. WebDAV range reads on encrypted files larger than one chunk returned zero or truncated bytes; the final chunk happened to work because the Telegram backend clamped at EOF. Switch the wrapper to the inclusive convention end-to-end: * `last_chunk = plaintext_offset_to_chunk(end, chunk_size)` * `trim_total = end - begin + 1` * normalise out-of-range / "to EOF" requests to `plaintext_total - 1` * pass `ct_end_excl - 1` to the inner repo so the on-wire byte count matches a whole number of GCM chunks Also expose the plaintext size to WebDAV clients: add an overridable `IFileContentRepository.content_length(fv)` (default: `fv.size`), override it in `EncryptingFileContentRepository` to subtract the file header and per-chunk overhead, and wire `Resource.content_length()` to call through to `Client.fc_repo`. HEAD requests and Content-Range now report the plaintext byte count instead of the ciphertext size. Tests updated to the inclusive convention; the in-memory mock now also uses `end - begin + 1` so it matches the real backend. 607/607 pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The encryption feature added `cryptography` and `argon2-cffi` to pyproject.toml but the lock file was not regenerated, so a clean `poetry install --no-root` against this repo failed with "pyproject.toml changed significantly since poetry.lock was last generated". Regenerated with `poetry lock` (no version bumps for existing direct deps). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The README mentioned ``master.salt`` had to be backed up but never said where it comes from. Document that TGFS auto-generates it on first start, where it gets written relative to ``TGFS_DATA_DIR``, how to seed it manually, and why it must never be rotated in place.
Add ``telegram.delete_messages_on_remove`` (default ``false``). When enabled, removing a file or directory via TGFS also issues ``messages.deleteMessages`` for the backing channel messages -- the file-descriptor message plus every content message of every version -- instead of leaving them on the channel as historical behavior does. Deletes are best-effort: failures are logged and never roll back the metadata change, and the bot must be admin of the file channel with "Delete Messages" permission for the API call to succeed.
Add ``telegram.delete_messages_on_remove`` (default ``false``). When enabled, removing a file or directory via TGFS also issues ``messages.deleteMessages`` for the backing channel messages -- the file-descriptor message plus every content message of every version -- instead of leaving them on the channel as historical behavior does. Deletes are best-effort: failures are logged and never roll back the metadata change, and the bot must be admin of the file channel with "Delete Messages" permission for the API call to succeed.
Directories used to share a hardcoded `created_at_timestamp` of `FIRST_DAY_OF_EPOCH`, so WebDAV clients displayed 1970-01-01 for every folder while files showed real dates. Store `created_at` and `modified_at` on `TGFSDirectory`, persist them through the serialized form (`createdAt` / `modifiedAt` in ms, optional fields so old metadata blobs deserialize unchanged), and expose `modified_at` as WebDAV `getlastmodified`. `modified_at` is bumped on the directly affected directory whenever its contents change (file/subdir added or removed) — POSIX mtime semantics, no cascade up the tree. Legacy directories saved without timestamps continue to read back at the epoch until an organic write pushes a fresh timestamp.
Adds created_at / modified_at timestamps to TGFS directories so WebDAV clients display real dates for folders. See upstream PR.
Adds opt-in deletion of channel messages when a file or directory is removed. See upstream PR.
- Type to_dict() of TGFSFileRef and TGFSDirectory with their TypedDicts so callers (including the round-trip test) pass the right type. - Refactor _read_ts to accept an int directly; with literal keys mypy can infer the value type from TGFSDirectorySerialized.
Replace internal-invariant asserts with explicit RuntimeError raises so they survive optimized bytecode (python -O) and stop tripping the ruff S101 lint. Also drop the unused imports that ruff's autofix flagged in stream.py.
Type serialization for directory and file reference models
Add message deletion support when removing files/directories
The TGFS encryption module already exists in the Python backend (``EncryptionConfig``) but the config-generator on the docs site had no UI for it, so users had to hand-edit ``encryption:`` into config.yaml. This adds a new ``EncryptionField`` covering every option from ``EncryptionConfig``: the enable toggle, the three mutually-exclusive passphrase sources (inline / env var / file), master_salt_file and chunk_size. Only the active passphrase source is emitted to the YAML, matching the loader's first-match semantics in tgfs/config.py. The form also surfaces the operational pitfalls (back up the salt, prefer env/file over inline) and a short explainer of the key hierarchy so first-time operators understand what they are turning on.
Pulls in the ``encrypt_names`` option that the encrypt-filenames branch adds to ``EncryptionConfig``. When enabled (default off), every uploaded document name -- including the pinned metadata blob -- is replaced with an AES-GCM ciphertext token (``TGFS1_<base64url>``), so a passive observer of the channel cannot read file or directory names from the document metadata. The UI exposes it as a sub-toggle inside the encryption block (it only applies when the master switch is on) and notes that the change is forward-only: existing parts in the channel keep their original names.
Add at-rest encryption configuration UI to config generator
Aligns Dockerfile.frontend and the GitHub Pages workflow on Node 24, the current Active LTS, replacing the EOL Node 18 base image and the older Node 20 in CI.
Bumps @types/node to ^24 so type definitions match the Node 24 runtime used in Docker and CI, and pins engines.node to >=24 so local installs warn on incompatible versions.
Upgrade Node.js version requirement to 24
…#2) * feat(crypto): opt-in encryption of telegram document names A new ``encryption.encrypt_names`` config flag replaces every Telegram-visible document name with an AES-256-GCM ciphertext token (domain-separated name key derived via HKDF from the master key). Plaintext names continue to live inside the already-encrypted metadata.json blob, so WebDAV and the manager UI stay unaffected while a passive observer of the channel no longer reads file or directory names from Telegram document metadata. Only new uploads are obfuscated; pre-existing files keep their original document name in the channel. * test(crypto): assert encrypted name stays consistent across multipart split * docs(encryption): explain how to set TGFS_MASTER_PASSPHRASE across deployments --------- Co-authored-by: Claude <noreply@anthropic.com>
) * fix(github-meta): restore real directory timestamps from git history The github_repo metadata backend rebuilds the directory tree from the repo's folder structure on every load and constructed each GithubDirectory without timestamps, so created_at/modified_at fell back to datetime.now(). WebDAV therefore reported the server-start time as every folder's creationdate, resetting on each restart. Recover the real dates from the repo's git history: created_at from the commit that introduced the directory's immutable `.gitkeep` placeholder, modified_at from the newest commit under the directory path. Falls back gracefully (keeps the default) when a path has no history, and never lets a history lookup crash the metadata load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(github-meta): never let timestamp lookup abort the directory load Broaden the commit-date lookup to swallow any error (not just IndexError/GithubException) so a failed history query degrades to the default timestamp instead of aborting the whole metadata rebuild. This also fixes the existing github_repo tests, whose mocked get_commits() returned a non-subscriptable Mock. Add a test asserting created/modified are restored from commit dates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The root has no .gitkeep to derive a creation date from, so it still fell back to now(). Use the backing repository's own created_at / pushed_at timestamps instead. Best-effort with an isinstance guard so it never sets a bogus value or breaks the load. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…epo (#8) With the github_repo backend the whole directory tree and all filenames were stored as plaintext folders/files in the GitHub repo. When encrypt_names is enabled we now also encrypt those path components, so the metadata repo no longer reveals the structure. - New tgfs/crypto/path_names.py: deterministic AES-256-SIV name encryption (TGFSP1_ prefix, base64url). Deterministic — unlike the random-nonce document-name scheme — because path components must round-trip to a stable storage identifier. - client.py derives a domain-separated path-name key from the master key and passes it to the github backend when encrypt_names is set. - gh_directory stores encrypted segments on write (dirs and file refs) and tracks per-directory whether it is stored encrypted, so encrypted and legacy-plaintext entries coexist (mixed read/write); delete tries the encrypted then the plaintext name. - The loader decrypts names on read and keeps using the storage path for the git-history date lookup, so the directory-timestamp fix still works. - Tests for the crypto module and the backend round-trip / mixed-read. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(scripts): add metadata path-name migration tool One-off, idempotent migration that re-encrypts existing plaintext directory/file names in the GitHub metadata repo to match what the server writes once encrypt_names is enabled. Defaults to a dry run; --apply rewrites the tree in a single commit after creating a pre-name-encryption backup tag. Run it only after deploying the name-encryption feature. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(scripts): allow print() in the migration CLI tool (ruff T201) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#10) _delete_github_directory used per-entry delete_file, which can only remove files (GitHub returns 422 on a subtree) and never recursed, so deleting a folder with subfolders left the nested .gitkeep/files behind — the folder kept reappearing in the repo (and after a reload, in listings). Rewrite the git tree without any blob under the directory's path and commit it once via the Git Data API, dropping arbitrarily-nested content atomically. Update the two affected tests. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in, transparent at-rest encryption layer that sits between
the file content API and the Telegram storage backend. When enabled, every
byte uploaded to Telegram is encrypted client-side -- the channel and the
metadata store only ever see ciphertext. Disabled by default; behavior is
unchanged for existing deployments.
Design
tag. Random-access decryption (HTTP Range, video streaming) keeps working.
HKDF-SHA256
the first Telegram message. A file can be decrypted from the channel
alone even if the TGFS metadata store is lost.
the header; flipped bits or chunk reordering are caught before plaintext
is returned.
IFileContentRepository; therest of TGFS is untouched. Adds
IFileContentRepository.content_length()so HEAD / Content-Range report the plaintext size, not the ciphertext
size.
Configuration
New optional
tgfs.encryptionblock (off by default):master_salt_fileis auto-generated on first start (16 random bytes,chmod 0600) relative to
TGFS_DATA_DIR. The salt is not secret but must bebacked up alongside the passphrase -- losing it makes re-deriving the
master key impossible. Full docs in the updated
README.mdanddemo-config.yaml.Backwards compatibility
byte-identically; no migration required.
on. Reads sniff the first four bytes: a
TGFSmagic routes through thedecrypt path, anything else is streamed back from the inner repo
unchanged. Writes always encrypt while the wrapper is active, so
overwriting a legacy plaintext file produces ciphertext.
TGFSbut whose header is truncated / malformed / fails MAC verificationraises
InvalidHeaderErrorrather than silently falling back toplaintext, which would mask a wrong master key.
Dependencies
cryptography(AES-GCM, HKDF, HMAC)argon2-cffi(Argon2id master KDF)Both are pinned in
pyproject.toml;poetry.lockis regenerated. Noversion bumps for existing direct dependencies.
Test plan
80 new tests added (49 unit + 18 end-to-end + 13 legacy-passthrough);
607/607 total pass, including all 487 pre-existing tests.
empty files and files smaller than one chunk)
matches Telegram backend's
bytes_to_read = end - begin + 1)TGFS-prefixed plaintext fails loudlyCommits
feat(crypto): transparent at-rest AES-256-GCM encryption-- core layerfeat(crypto): read legacy plaintext files transparently-- migrationfix(crypto): use inclusive end ranges; expose plaintext content-lengthbuild(deps): refresh poetry.lock for cryptography and argon2-cffidocs: explain how the encryption master salt is created