Please update or remove your doicumentation at bcachefs-docs.readthedocs.io - #1
Open
noradtux wants to merge 9596 commits into
Open
Please update or remove your doicumentation at bcachefs-docs.readthedocs.io#1noradtux wants to merge 9596 commits into
noradtux wants to merge 9596 commits into
Conversation
The btree cache cannibalize lock is an open-coded mutex held by one thread per fs while it grows the btree cache by evicting other entries. The deadlock-trigger we hit in CI (replicas_variable_buckets hung-task, 30s allocator stall) is: thread A holds cannibalize (via bch2_btree_reserve_get) thread A's __bch2_btree_node_alloc calls bch2_alloc_sectors_req allocator is out of buckets, closure_wait, trans_unlock, sleep reclaim wants to free buckets but needs cannibalize to grow cache -> circular wait, no forward progress lockdep can't see this because the cannibalize lock is an open-coded mutex (bc->alloc_lock = current), not a tracked mutex. Auditing the four call sites (interior.c:622, interior.c:3557, read.c:1090, iter.c:1227): only interior.c:622 and iter.c:1227 hold cannibalize across an operation that can sleep, and in both cases the sleep happens via trans_unlock. Dropping cannibalize on trans_unlock means it's only ever held over non-sleeping critical sections. The resource deadlock can't close because by the time we're sleeping in the allocator we no longer hold cannibalize, so reclaim can take it and make progress. Callers that need cannibalize after a wake re-acquire normally (the existing err/retry paths already handle the not-held case via the bc->alloc_lock == current check inside the cache code). A bitfield bool on btree_trans tracks whether this trans holds cannibalize, so the common path through bch2_trans_unlock doesn't have to touch the shared bch_fs cacheline. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
…failure bch2_setattr_nonsize() transfers in-memory quota counters from the inode's old qid to the new qid before calling the btree update that actually persists the new qid on the inode. bch2_fs_quota_transfer() also updates inode->ei_qid in place. If the subsequent lockrestart_do(bch2_setattr_nonsize_trans) returns a real error (ENOSPC, etc., as opposed to a transaction restart which is retried internally), the in-memory state has been mutated but the on-disk inode still has the old qid. Subsequent operations on the inode use the (now-divergent) in-memory ei_qid for accounting, so the drift compounds. generic/270 stresses this with fsstress -fsetattr=1 running concurrent chowns against a dd-to-ENOSPC workload; the test catches the divergence via a quota report vs. inode-scan comparison. Save the old qid, run the commit, and on real-error roll the transfer back to the saved qid. NOCHECK on the rollback because we're moving counters back to a qid they just came from - it had room. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
bch2_set_folio_dirty() WARN_ON'd if folio_pos(folio) >= i_size, but that condition is reachable along race paths the VFS truncate contract explicitly tolerates. truncate_setsize() drops i_size before calling truncate_pagecache(), so between the two there's a window where the folio is fully attached to the address_space with i_size already small. Folios pinned by gup (e.g. DIO read destinations into an mmap'd region of this file) can also persist after truncate_pagecache, since truncate_inode_pages can't fully evict pinned folios - they stay in the cache with their dirty bit handled lazily. The VFS handles this via truncate_cleanup_folio()'s folio_cancel_dirty() ordering (the comment there explicitly notes "Some filesystems seem to re-dirty the page even after the VM has canceled the dirty bit"). XFS's iomap_dirty_folio and ext4_dirty_folio both dirty unconditionally and trust the cancel ordering. bcachefs already cleans up the per-sector dirty bits in __bch2_writepage when it sees above-i_size data, per the comment that remains. Drop the WARN; this is a race the codebase already handles, not an invariant violation. generic/503 (mmap collision with DIO read into mmap'd file) triggers this from bio_set_pages_dirty in the DIO read submit path. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Explain why the PHYS field exists on backpointers (it's a denormalized index for cheap fsck pairwise checks against reconcile_phys, avoiding expensive random lookups from reconcile_phys to extents), and state the invariant: every path mutating reconcile_opts on an extent must keep the bp's PHYS flag in sync with the reconcile_phys btree. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
bch2_do_discards() walks the need_discard btree submitting a REQ_OP_DISCARD bio per bucket. discard_endio() marks the in_flight entry complete and drops the per-device refcount, but the entry itself is only ever freed by bch2_discards_complete(), which the loop only invoked on -BCH_ERR_max_discards_in_flight. When discards complete inline - the device doesn't support REQ_OP_DISCARD (BLK_STS_NOTSUPP fires bio_endio() from submit context), or it's fast enough that we never reach DEV_IN_FLIGHT_MAX - the per-device refcount returns to zero immediately, so we never take that path and the in_flight darray grows without bound across the btree walk. The linear darray_find_p() scans in discard_in_flight_add() (twice per bucket) and discard_endio() then make the walk O(n^2): a discard kworker pinned at 100% CPU. Reap completed discards as we go - in_flight.nr exceeding the in-flight bio count means there are entries waiting to be reaped - advancing the need_discard iterator before the nested restart so we don't reprocess the bucket. Also skip the bio dance entirely when bdev_max_discard_sectors() is zero: the device doesn't support discards, so just mark the bucket free. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
… indirect extent A non-degraded data extent carries no bch_extent_reconcile entry: the reconciler re-derives the file's IO options from the inode on each pass, so there's nothing to record. But an indirect extent has no inode to derive from - bch2_bkey_get_io_opts() for KEY_TYPE_reflink_v falls back to the filesystem defaults unless the extent carries a bch_extent_reconcile entry of its own. So when bch2_make_extent_indirect() copies a data extent's value into a new reflink_v, if the source extent had no reconcile entry (because it matched its inode's options), the reflink_v ends up with none either, and the reconciler reconciles it against the filesystem defaults rather than the inode's options. If those differ - a per-inode data_replicas, a background_target - reconcile does the wrong thing: e.g. it tries to bring every freshly-reflinked extent to the default replica count, which can ENOSPC the filesystem. Snapshot the source inode's IO options onto the new reflink_v with bch2_bkey_set_needs_reconcile(): bkey_should_have_rb_opts() has a KEY_TYPE_reflink_v case that carries the entry whenever an option is set from the inode (not only when need_rb is set), so this creates the entry exactly when the inode has non-default options, and is a no-op otherwise. Reserve buffer headroom for the entry plus BCH_SB_MEMBER_INVALID padding, matching the data update path. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
__bch2_truncate_folio() WARN_ON_ONCE'd if folio_pos(folio) >= i_size,
but that's not an invariant we maintain: marking a folio dirty doesn't
go through i_rwsem, so a folio wholly above i_size can be present in the
mapping even with the inode fully locked. The standard offender is
bio_set_pages_dirty() off the DIO read completion path - a DIO read
whose destination buffer is an mmap of this file, completing after the
file was truncated down - which is the same race eb56878d63d4
("bcachefs: bch2_set_folio_dirty(): drop over-strict above-i_size
WARN") removed the sibling WARN for.
Hit from fsstress via bch2_fallocate_dispatch -> bchfs_fpunch ->
bch2_truncate_folios -> __bch2_truncate_folio, when fpunch's range
overlaps such a leftover folio (filemap_lock_folio() finds it).
The code right below already copes: when the folio is wholly above
i_size, end_pos stays at folio_end_pos() (in bounds), and
__bch2_writepage cleans up the per-sector dirty bits when it sees
above-i_size data. XFS's iomap_dirty_folio and ext4_dirty_folio both
dirty unconditionally and trust the truncate_cleanup_folio() cancel
ordering rather than asserting this. Drop the WARN; keep a comment.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
…s_sorted.nr alloc_request_get() uses bch2_trans_kmalloc_nomemzero() (struct alloc_request is large - two open_buckets, two bch_devs_mask, the trace DARRAY... - a full memset on the alloc fast path isn't free) and explicitly initializes the fields it sets, but was missing three: nr_effective, devs_sorted.nr, devs_may_alloc. bch2_alloc_sectors_req() populates all three before anything reads them, so the normal write path was fine - but bch2_bucket_alloc_trans() callers that go through it directly (bch2_set_nr_journal_buckets_iter(), i.e. `bcachefs device resize-journal`) only set req->ca, leaving the rest as whatever junk the bch2_trans_kmalloc slot held. The garbage nr_effective made req_alloc_should_bail()'s "have_replicas" check true on a fresh request, so when the device was momentarily out of free buckets the allocator returned bucket_alloc_no_progress instead of parking on freelist_wait - and bch2_set_nr_journal_buckets_iter() only retries via bch2_wait_on_allocator() on operation_blocked, so it bailed out with -EAGAIN. (devs_may_alloc was the visible-garbage one in the alloc-request dump; it's only read by bch2_alloc_request_to_text() in that path, but zero it too - it's a foot-gun for any future direct bch2_bucket_alloc_trans() caller.) Reported by the journal_resize ktest. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
An opt change is a multi-step operation - it brackets the actual change with scan-cookie bumps (see opts.c) so writers' extent triggers re-derive against the new option as it lands. But the reconcile thread must not *complete* (delete) a scan cookie for a scan pass that overlapped a half-applied opt change: such a pass scanned against the intermediate option value, and on the emergency-read-only / error path nothing bumps the cookie afterwards to force another pass - so the extents (and the denormalized RECONCILE_PHYS flags on their backpointers) are left stale, with no pending scan to fix them. Add an rhashtable of refcounted in-flight scan cookies, registered for the duration of an opt change via bch2_set_reconcile_needs_scan_pre() / _post(). bch2_clear_reconcile_needs_scan() refuses to delete a cookie that's registered; the reconcile thread also skips starting a scan it couldn't complete anyway, but that's just sparing wasted work - the don't-delete is what's load-bearing. The mutex serializes the refcount transitions; the reconcile thread's "is this cookie in flight?" check is a lockless rhashtable lookup, so entries are freed via kfree_rcu(). This is just the mechanism: _pre()/_post() aren't wired into the opt-change path yet - that, with a guaranteed pre<->post pairing on the error path, is a follow-up - so the in-flight set is empty for now and the clear-side check is a no-op. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
…_io() Hook bch2_set_reconcile_needs_scan_pre()/_post() into the bracketed reconcile scans in opt_hook_io() (the fs / inum / metadata / device / stripes scans - not the durability "re-look at pending" one), so the reconcile thread won't complete (delete) one of those scan cookies while the opt change that touched it is mid-flight. That closes the change_replicas backpointer staleness: previously the reconcile thread could pick up the pre-set scan cookie, scan + drop bch_extent_reconcile against the intermediate option value, and clear the cookie - and on the ERO/error path nothing re-armed it, leaving the denormalized RECONCILE_PHYS flags on those extents' backpointers pointing into the reconcile_phys btree at entries that no longer exist. The pre<->post pairing has to hold even when the opt change errors out or never reaches bch2_opt_hook_post_set() (the sysfs path skips it when the value didn't actually change; the xattr path can fail the inode write between pre and post; ERO can't commit the post-set bump at all) - otherwise the in-flight registration leaks and wedges reconcile on that cookie forever. Rather than auditing every caller to always-pair, give the registration a lifetime: bch2_opt_hook_pre_set() takes an opt_change_scope, the caller holds it as a CLASS() across the pre->change->post span, and the destructor unregisters whatever was registered. The scope starts empty, so a change that never reaches pre_set (or fails before registering) destructs harmlessly. bch2_set_reconcile_needs_scan_post() therefore no longer drops the in-flight registration - the scope destructor owns it. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
00677 BUG: sleeping function called from invalid context at include/linux/percpu-rwsem.h:51
00677 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 34081, name: xfs_io
00677 preempt_count: 1, expected: 0
00677 RCU nest depth: 1, expected: 0
00677 3 locks held by xfs_io/34081:
00677 #0: ffffff80c34d0a10 (&mm->mmap_lock){++++}-{4:4}, at: exit_mmap+0x68/0x480
00677 toppk#1: ffffffc0811b0eb8 (rcu_read_lock){....}-{1:3}, at: ___pte_offset_map+0x8/0x1a8
00677 koverstreet#2: ffffff80c3321a98 (ptlock_ptr(ptdesc)){+.+.}-{3:3}, at: __pte_offset_map_lock+0x80/0x160
00677 Preemption disabled at:
00677 [<ffffffc0802457a8>] __pte_offset_map_lock+0x80/0x160
00677 CPU: 3 UID: 0 PID: 34081 Comm: xfs_io Not tainted 6.17.0-ktest-g1051044265e5 #54360 PREEMPT
00677 Hardware name: linux,dummy-virt (DT)
00677 Call trace:
00677 show_stack+0x1c/0x30 (C)
00677 dump_stack_lvl+0x68/0x90
00677 dump_stack+0x14/0x1c
00677 __might_resched+0x170/0x270
00677 __might_sleep+0x4c/0x90
00677 percpu_down_read_internal.constprop.0+0x24/0x148
00677 __bch2_disk_reservation_add+0x5c/0x460
00677 bch2_disk_reservation_add+0xe4/0x198
00677 bch2_vfs_dirty_folio+0x1a0/0x2d0
00677 folio_mark_dirty+0x38/0xa0
00677 unmap_page_range+0xe64/0x1100
00677 unmap_single_vma.isra.0+0x88/0xe8
00677 unmap_vmas+0x68/0x138
00677 exit_mmap+0xb4/0x480
00677 mmput+0x7c/0x180
00677 do_exit+0x23c/0xab8
00677 do_group_exit+0x38/0xa0
00677 get_signal+0xaa4/0xad0
00677 do_signal+0x84/0x250
00677 do_notify_resume+0xf8/0x160
00677 el0_svc+0xa4/0xb0
00677 el0t_64_sync_handler+0x98/0xe0
00677 el0t_64_sync+0x154/0x158
mark_lock dated from when the percpu allocation online_reserved was on
could be resized - it's no longer necessary, and sectors_available_lock
can safely be a spinlock.
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
… hot path
CONFIG_INIT_STACK_ALL_PATTERN=y (default on most kernel configs) makes
the compiler initialize every automatic variable to 0xFE on scope
entry. bch2_bkey_cmp_packed_inlined() declared
struct bkey unpacked;
at function scope, so every call to it - i.e. every sort/merge/
heap-pop compare in bch2_sort_keys() and friends - emitted 5 movabs
stores filling sizeof(struct bkey) == 40 bytes of stack with the
poison pattern, on the hot likely(both-packed) branch where the
buffer is never touched.
Move the declaration into the cold mixed packed/unpacked branch where
the buffer is actually used. The auto-init follows scope, so it now
only runs when we are about to unpack one side anyway.
sort.o disasm (gcc CONFIG_INIT_STACK_ALL_PATTERN=y x86_64):
before: function prologue includes
sub $0x78,%rsp ; 120 byte frame
mov %rax,0x28(%rsp) ; 0xFEFE... x 5
mov %rax,0x30(%rsp)
mov %rax,0x38(%rsp)
mov %rax,0x40(%rsp)
mov %rax,0x48(%rsp)
after: hot path is endbr64; format-byte test; ja cold; the
stack frame and poison stores now live only in the cold
mixed branch. Frame on the cold path drops to 0x60 == 96
bytes (no unpack slot for the both-packed case).
Saves 5 stores + smaller frame on every comparison; cmp is the inner
of bch2_sort_keys() and the eytzinger aux-tree binary search, so the
per-call cycles add up. Hot-path observation from Kent looking at
the inlining around kwz's sqlite-bench vtune profile on #bcachefs-ai.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Single command to build the kernel module via DKMS and load it on the running host. Idempotent — re-running rebuilds + reloads. Must be run as root (sudo make dkms-reload, or directly in a CI VM). Wraps the existing install_dkms staging with dkms add/build/install + modprobe -r/modprobe, plus a final modinfo for visibility. This is the gap that's been making it awkward to test bcachefs-tools commits in CI the way we test kernel commits today. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Empty sections (after the count>0 filter trims everything) were getting skipped entirely in the TUI, so you couldn't tell from looking whether the section had no entries or had entries that all happened to be silent. Always emit the section label + column header; an empty body just means "this category is quiet right now," which is itself useful info during perf work. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The mutex-protected list of bcachefs inodes was a contention point on every inode load/evict and a serialised walker for the snapshot create unmap path. Switch to fast_list: lockless add/remove off a percpu slot buffer, and lockless iteration backed by a genradix. - pre-allocate the slot index in __bch2_new_inode (the gfp-aware path) so the hash-insert success branch is failure-free; release in the error paths that bypass evict. - bch2_evict_inode releases the slot via fast_list_remove (which is a no-op when ei_inodes_idx == 0, covering the discard_new_inode race). - bch2_evict_subvolume_inodes switches to rcu_read_lock + fast_list_for_each; inodes are RCU-freed via .free_inode, so the pointer is stable until igrab pins or we drop rcu_read_lock. - bch2_unmap_mapped_inodes in vfs/ioctl.c (snapshot create unmap path) uses the same fast_list walker instead of the prior sb-wide s_inode_list_lock walker. inode_state_read* shims move from fs.c to fs.h so ioctl.c can use them. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
fast_list_iter tracks position by integer index, so we can drop rcu around the dcache work (held safe by the igrab ref) and resume from the same slot on the next iteration — no need to collect inodes into a batch and drain in a second pass. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Snapshot creation took c->snapshots.create_lock as a writer but nothing took it as a reader, so it only serialized snapshot creations against each other — not against concurrent IO. sync_inodes_sb in the snapshot ioctl flushes existing dirty pages before the snapshot transaction, but between that flush and the actual snapshot transaction running there's a window (path lookup, permission checks, security_path_mkdir) where new pages can be dirtied. If the kernel's async writeback then picks up some of those new dirty pages but not all of them before the snapshot transaction, the snapshot captures a partial state. mclaud hit this with MySQL + nosync.so + hourly snapshots: InnoDB data pages with LSN-in-the-future relative to the system LSN, classic "data page flushed but the corresponding redo log page wasn't" snapshot-captures-half-of-a-write shape. The fix is what the original lock was clearly intended to be: take it read-side on every path that dirties the page cache. O_DIRECT and buffered writeback don't need it (their btree transactions are already atomic w.r.t. snapshot creation); buffered write_iter and mmap page_mkwrite do. Convert create_lock to percpu_rw_semaphore for fast read-side. This is fine while all read-side acquire/release is in the same task — which is true for synchronous buffered writes and page faults today. If async buffered IO is ever added (e.g. io_uring buffered via worker threads), the cross-task release will break percpu_rwsem's lockdep tracking; comment in bch2_write_iter calls out the switch needed at that point. Tested-by: qubitnano <146656568+qubitnano@users.noreply.github.com> Co-Authored-By: Proof of Concept <poc@bcachefs.org> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
Two bugs that combine to drop nearly all DKMS sources on disk: 1) VERSION was recursively-expanded (`=` not `:=`), so `$(shell git describe)` re-ran on every $(VERSION) expansion. The install_dkms recipe references $(DKMSDIR) — which expands $(VERSION) — six times, one per install line. If HEAD moves mid-recipe (a commit or rebase landing in another shell, which is the normal case during long sessions), $(DKMSDIR) resolves to different paths within a single recipe execution. The result was an incomplete tree under each: the first `install -D` commands populated /usr/src/bcachefs-v<A>/, the trailing ones populated /usr/src/bcachefs-v<B>/, neither was complete, and `dkms add` then picked one and silently saw e.g. just module-version.c + version.h with no Makefile, no fs/, no dkms.conf. Symptom Kent saw: `ls /usr/src/bcachefs-v<hash>/src/ fs/bcachefs/` showed only module-version.c + version.h. Lock VERSION (and VERSION_H, same shape) once at make start. Six git describe shell-outs were also a measurable cost on top. 2) The trace-header sed pattern was stale after the fs/bcachefs/ → fs/ restructure: trace.h moved into a debug/ subdir and its TRACE_INCLUDE_PATH gained a /debug suffix (`../../fs/bcachefs/debug`). The sed pattern still matched the pre-restructure form (`../../fs/bcachefs`) so it no-op'd, leaving the kernel-tree-relative path baked into the DKMS source — kbuild then couldn't find the trace headers at module-build time. Verified end-to-end via `make install_dkms DESTDIR=/tmp/dkms-test`: six install lines now land in the same DKMSDIR, all 306 fs/*.[ch] files present, and trace.h's TRACE_INCLUDE_PATH is rewritten to `.`. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Journal reclaim isn't allowed to run ahead of journal replay, to avoid consuming space in the journal that's still pinned by replay. That means transaction commits can't block on journal reclaim if we don't want to deadlock.
vtune profiling of bcachefs FUSE running sqlite-bench (10k entries, WAL, tmpfs) found __bkey_unpack_pos consuming 7.5% of all retired instructions - 5.05B Ir under cachegrind, 0.211s vtune - via get_inc_field()'s state machine: ~25 insns/field x 3 fields per call. The full-key path (__bch2_bkey_unpack_key_b) already has a byte-aligned fast path using @b->unpack[] and unpack_field_fast() (~3 insns/field), but the position-only path was still going through the 2004-era bit-stream extractor. Add __bkey_unpack_pos_b() mirroring the same gate (byte_aligned_fields) and precomputed @b->unpack[] constants, and route bkey_unpack_pos_format_checked() through it on builds without HAVE_BCACHEFS_COMPILED_UNPACK (i.e. userspace / bcachefs-tools FUSE, where the profile was generated). Falls back to __bkey_unpack_pos() on non-byte-aligned formats, matching the gating used by the full-key fast path. x86_64 disasm: ~80 insns of state-machine bit-stream extraction drop to ~25 insns (three load-shift-add field reads plus EBUGs/prelude). On the original sqlite-bench workload this should subtract roughly 5% of total Ir, with the bulk landing on the FUSE event-loop thread where bkey unpack was the second-hottest function after memset. Profile data and proposed fix shape from kwz on #bcachefs-ai. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Kill gotos, improve overflow check, improve error messages
The 64K cap was hit on a 34-device array's accounting tree (Zorba):
alloc_percpu: dynamic arena exhausted (used 65536, requested 8, max 65536)
accounting_read(): error ENOMEM_disk_accounting
Each thread chunk grows from 64K to 512K, but calloc()'d pages map to
the zero page until written, so untouched space costs address space,
not RAM. Real cost is the size_at_grain table (now 128K, one copy).
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
No real consumers and the cross build pulls in sqlite-i686 (via util-linux's transitive buildInputs), whose tcltest suite fails on i686. Drop the target instead of carrying the overlay that strips sqlite from util-linux for cross builds. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Levels under path->level should not point to valid btree nodes.
Shows how far back the journal can be safely rewound, and lists the flush entries within that window — each is a valid rewind target. Opens the filesystem in read_journal_only mode (same flags as list_journal), collects every journal entry, finds the most recent, and reads its BCH_JSET_ENTRY_rewind_limit sub-entry to determine the rewind floor (discards may have invalidated anything earlier). Then walks entries in [floor, latest], filters to flush entries, and prints (seq, datetime) for each, pulling the timestamp from each flush's BCH_JSET_ENTRY_datetime sub-entry. If the latest entry has no rewind_limit sub-entry (older filesystems where it was never written), falls back to the lowest seq present on disk with a warning. -n N trims output to the most-recent N flush candidates, since the operator usually cares about "what can I rewind to from close to now?". Placed under Repair (next to fsck), since that's the workflow. bch_bindgen doesn't expose jset_entry_datetime or jset_entry_rewind_limit; the sub-struct __le64 payload at offset 8 is read directly with read_unaligned(). If we end up needing those types in more places they should move into bch_bindgen. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
1411f73 started skipping completions whenever CARGO_BUILD_TARGET was set, which broke the Debian build: dh_install requires the completions file per debian/bcachefs-tools.install, but the Makefile no longer produces it. Debian's packaging passes --target x86_64-unknown-linux-gnu on x86_64 build hosts as a packaging convention - the binary is fully native and can generate the completions file. Restrict the skip to the case where the target arch actually differs from $(uname -m). Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Makefile:12 pipes \`cargo metadata\` through jq to compute VERSION:
VERSION:=$(shell cargo metadata --format-version 1 | jq -r ...)
The nix flake build environment didn't have jq, so the pipeline
silently produced an empty VERSION and the make parse downstream
failed with "recipe commences before first target."
Add jq to nativeBuildInputs of crane-build.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The previous fix put the SKIP_COMPLETIONS=1 variable assignment inside the install recipe body (between recipe lines), surrounded by ifdef/ifneq/endif. Some make versions terminate the recipe on the variable assignment, leaving subsequent recipe lines orphaned at top level - which triggers "Makefile:NNN: recipe commences before first target." Compute SKIP_COMPLETIONS at top level before the install target. The install body just uses ifdef SKIP_COMPLETIONS to pick between the SKIP message and the actual completions generation - which is the conditional pattern make is happy with mid-recipe. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
With fsck=1 the kernel ORs in the full PASS_FSCK default set on top of c->opts.recovery_passes, so a user passing -o recovery_passes= check_dirents would still run check_allocations, check_alloc_info, check_btree_alloc, check_lrus, etc. — the entire default offline fsck pipeline. Detect recovery_passes in cli.opts and skip the fsck option in that case, so the user's pass selection is what actually runs. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
Aim for 1/14th of system ram, instead of a hard 8GB limit.
Avoid stalling the journal on cache flushes.
Positions printed in dmesg and bcachefs_to_text use the literal tokens U64_MAX and U32_MAX as sentinel values - e.g. 258779:2660224:U32_MAX. Let bpos's FromStr accept those tokens per-field so positions can be pasted directly into tools that take a bpos argument. inode/offset (u64): accept U64_MAX, U32_MAX, or numeric. snapshot (u32): accept U32_MAX or numeric. A present-but-unparseable snapshot field is now a hard error rather than silently defaulting to 0; absent snapshot still defaults to 0. bbpos's FromStr and bbpos_range_parse inherit this through bpos. Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The top-level `make debug` target had broken syntax for the userspace
build (the -D tokens after `debug:` weren't variable assignments and
weren't real prerequisites) and never touched the DKMS module path. The
DKMS install captured fs/Makefile's existing BCACHEFS_DEBUG /
BCACHEFS_TESTS env-var hooks only if the user manually set them at
`dkms build` time on the target host.
Plumb a build-options file through install_dkms so the choice made at
userspace install time persists into the host-side module build:
- Top-level Makefile gathers any of $(BCACHEFS_DKMS_FORWARD)
(BCACHEFS_DEBUG, BCACHEFS_TESTS, BCACHEFS_INJECT_TRANSACTION_RESTARTS)
that are set and writes them as Make assignments to
$(DKMSDIR)/build.vars during install_dkms. Empty if none are set.
- dkms/Makefile -includes build.vars and re-exports the same names,
so the recursive kbuild inherits them and fs/Makefile's existing
ifdefs fire during the dkms-driven module compile.
- `make debug` is now a real alias: sets BCACHEFS_DEBUG=1 and
BCACHEFS_TESTS=1 for the whole invocation (so `make debug install_dkms`
propagates) and appends -DCONFIG_BCACHEFS_DEBUG=y -DCONFIG_VALGRIND=y
to EXTRA_CFLAGS for the userspace build. BCACHEFS_INJECT_TRANSACTION_RESTARTS
is intentionally not pulled in by `debug` - it belongs to its own
dedicated test variant.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
The verify check after btree_split / node_rewrite was firing on stale
cached pointers from a different btree, panicking copygc_torture:
path still references btree node being freed, level 1:
extents level 1/1 ... ptr: vdb 0:2927:352 gen 2
path: ... btree=backpointers level=0 ...
l=1 locks unlocked seq 11702 node ffffff80d73c4800
The lingering path was a bp_iter in BTREE_ID_backpointers, parked at
level 0. Its higher levels were unlocked but still carried the cached
node pointers from the descent — that's by design, for fast relock
without re-traverse (the six_lock seq check on relock catches if the
slot was reused, forcing re-descent).
When the btree-node-cache slot for one of those released-but-cached
backpointers internal nodes was evicted and reused for the extents
level-1 node we now wanted to free, path->l[1].b == b became true by
coincidence of cache memory reuse — though the bp_iter never touched
the extents btree.
bch2_trans_node_add migrates same-btree paths whose pos covers the new
nodes; the verify is checking that drain succeeded. So the assertion
that bch2_trans_node_add actually maintains is "no path still holds a
lock at this level pointing at b" — released-cached pointers are not
in scope. Tighten the check accordingly.
Co-Authored-By: Proof of Concept <poc@bcachefs.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs at https://bcachefs-docs.readthedocs.io/ are badly outdated and people continue stumbling over it, thinking this was the official docs for bcachefs. It seems to be multiple years old, lots of things in there are not valid any more. Official docs live here: https://bcachefs.org/bcachefs-principles-of-operation.pdf
(sorry, didn't know any other way to communicate this than to open a pull request)