Make ClockCache hits lock-free

CLAUDE.md

@@ -40,6 +40,7 @@ Keep Markdown prose to one sentence per line.
 - `@cached` preserves the method signature and moves the body to `MemoizationKit.implementation`.
 - The default global cache shares one size limit across a function's cached methods, with a separate cache per container type.
 - Preserve inferred return types and allocation-free RAM hits for concrete keys with the built-in containers.
+- `ClockCache` hits are lock-free; keep the rules listed in `src/containers/clock.jl`. Task-local caches use the locked `LocalClockCache`.
 - Built-in RAM containers distinguish key types and use `hash` and `isequal`.
 - `MemoizationKit.cachekey` may merge equivalent inputs; merged calls must accept the same result and return type.
 - Compute misses outside cache locks so recursion works; concurrent misses may compute twice.

benchmark/Project.toml

@@ -4,6 +4,7 @@ MemoizationKit = "1b238080-9255-4fe9-b224-89eb24efe93b"
 LRUCache = "8ac3fa9e-de4c-5943-b1dc-09c6b5f20637"
 Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 
 [sources]
 MemoizationKit = {path = ".."}

docs/src/eviction.md

@@ -34,6 +34,7 @@ Unlike LRU, Clock does not track the exact order of hits.
 A recently inserted entry that has never been hit again can be evicted before an older entry that has been reused.
 
 Choose Clock for approximate recency with little bookkeeping on hits: a hit updates a bit rather than a recency list.
+A shared `ClockCache` is read without a lock, so hits from many threads do not wait on each other.
 Eviction may scan several slots to find a victim.
 
 ## LRU: exact recency
@@ -59,7 +60,7 @@ sort!(collect(keys(cache))) # B was evicted
 Unlike Clock, LRU always retains the exact recency order, even after several hits between evictions.
 
 Choose LRU when that ordering suits the workload.
-Maintaining the order requires more bookkeeping on hits, but choosing the next victim does not require a scan.
+Maintaining the order requires more bookkeeping on hits, under a lock, but choosing the next victim does not require a scan.
 Neither policy guarantees a better hit rate for every workload; use the [dashboard](dashboard.md) or [timing](timing.md) to evaluate yours.
 
 ## Selecting a policy
@@ -83,5 +84,5 @@ Custom eviction policies can be supplied through the [cache interface](interface
 Both policies evict as many entries as needed to fit a new value or a reduced limit.
 A value larger than the entire limit is returned without being stored.
 Byte limits measure cached values, not keys or container overhead.
-Shared caches use locks under either policy; computations run outside those locks.
+Shared caches take a lock to insert and evict entries, and `LRU` also on hits; computations run outside those locks.
 Disk caches have no eviction policy; see [Disk caching](disk.md) for cleanup.

docs/src/index.md

@@ -167,6 +167,7 @@ See [Configuration](configuration.md) for persistent defaults and custom size me
   Changes to external state or method definitions (including through Revise) do not automatically invalidate results.
   Clear the global cache with `empty_caches!`; for disk results, update [`MemoizationKit.diskversion`](@ref).
 - Built-in shared caches are thread-safe.
+  [`ClockCache`](@ref) hits take no lock, so they scale with the number of threads.
   Computations run outside the cache lock, allowing recursion; simultaneous misses may compute the same key more than once.
   Exceptions are propagated without storing a result.
 
@@ -183,7 +184,7 @@ Strategies, settings, persistence, and monitoring work together under one functi
 | [Memoize.jl](https://github.com/JuliaCollections/Memoize.jl#usage) | You want a small `@memoize` API with a dictionary of your choice. | Limits and statistics depend on the container; persistence, a dashboard, and package settings require additional integration. |
 | [Memoization.jl](https://github.com/marius311/Memoization.jl#readme) | You need to memoize individual calls or closures, as well as method definitions. | Its default cache is unbounded and not thread-safe; custom containers can address those needs, but management and persistence tools require additional integration. |
 | [LRUCache.jl](https://github.com/JuliaCollections/LRUCache.jl#readme) | You want a thread-safe cache dictionary with limits, statistics, resizing, and eviction callbacks. | You write the function wrapper and management code, or combine it with a memoization macro. |
-| MemoizationKit.jl | You want bounded caches, per-type strategies, disk persistence, and live monitoring under one API. | The macro requires a method definition; shared caches use locks, and disk results need explicit versioning and cleanup. |
+| MemoizationKit.jl | You want bounded caches, per-type strategies, disk persistence, and live monitoring under one API. | The macro requires a method definition; shared caches lock to insert entries (and `LRU` also on hits), and disk results need explicit versioning and cleanup. |
 
 Memoize.jl and Memoization.jl both accept LRUCache.jl containers, so bounded memory and statistics are available without MemoizationKit.
 Memoization.jl also preserves return-type inference.

src/MemoizationKit.jl

@@ -26,6 +26,7 @@ include("containers/interface.jl")
 include("containers/slots.jl")
 include("containers/lru.jl")
 include("containers/clock.jl")
+include("containers/localclock.jl")
 
 include("registry.jl")
 include("preferences.jl")

src/cachestyle.jl

@@ -54,7 +54,9 @@ Strategy that stores results in caches local to the current task, of container t
 `C` is either an `AbstractCache` or an `AbstractDict` type such as `Dict`; if it is not
 already concrete, it is completed to `C{K, V}`. Without `C`, the container is the one set by
 the `container` preference. Task-local caches need no locking but are not shared, not bounded
-unless `C` is, and are not visible to [`cache_info`](@ref).
+unless `C` is, and are not visible to [`cache_info`](@ref). `TaskLocalCache{ClockCache}` uses
+an internal variant of [`ClockCache`](@ref) whose hits take the lock, which costs nothing
+uncontended and saves the memory of the per-thread counters.
 """
 struct TaskLocalCache{C} <: CacheStyle end
 

src/call.jl

@@ -80,7 +80,10 @@ end
 
 # One untyped cache per function for the containers of MemoizationKit; other dictionaries (e.g. `Dict`)
 # would box untyped keys, so they get one typed dictionary per key and value type.
-_localtype(C, K, V) = C isa UnionAll ? (C <: AbstractCache ? C{Any, Any} : C{K, V}) : C
+_localtype(C, K, V) = C isa UnionAll ? (C <: AbstractCache ? _localcache(C){Any, Any} : C{K, V}) : C
+# A task-local cache is used by one task only, so lock-free hits gain nothing there, while the
+# per-thread counters of `ClockCache` would cost memory in every task.
+_localcache(C) = C === ClockCache ? LocalClockCache : C
 # Task-local caches follow the size settings of `f`, but are not registered.
 function _newlocal(f, ::Type{T}) where {T <: AbstractCache}
     fc = @lock REGISTRY.lock _functioncaches!(f)

src/containers/clock.jl

@@ -1,3 +1,36 @@
+# An entry of a `ClockCache`; only its reference bit changes after construction.
+mutable struct Entry{K, V}
+    const hash::UInt # the fields read on a hit come first
+    const key::K
+    const val::V
+    @atomic used::Bool # the reference bit
+    const size::Int
+    const position::Int # in `ring`
+end
+
+struct Tombstone end
+const TOMBSTONE = Tombstone()
+
+# The table of entries is an open-addressing hash table with linear probing, at most half full,
+# read without the lock. This is safe because:
+# 1. entries do not change after construction, apart from their reference bit;
+# 2. a published table never changes length, but is replaced as a whole;
+# 3. a slot only changes, under the lock, from `nothing` or `TOMBSTONE` to an entry, from an entry
+#    to `TOMBSTONE`, or to `nothing` if the next slot is `nothing`. Entries never move, so a probe
+#    finds every entry that is present.
+# Slots are loaded with acquire and stored with release ordering. Before Julia 1.12, which added
+# atomic indexing, `setindex!` into a `Vector{Any}` is a release store with a GC write barrier
+# (checked by the tests); storing through a pointer would skip the barrier.
+@static if VERSION >= v"1.12"
+    const SlotTable = AtomicMemory{Any}
+    loadslot(t::SlotTable, i::Int) = @atomic :acquire t[i]
+    storeslot!(t::SlotTable, i::Int, x) = (@atomic :release t[i] = x; t)
+else
+    const SlotTable = Vector{Any}
+    loadslot(t::SlotTable, i::Int) = GC.@preserve t unsafe_load(pointer(t, i), :acquire)
+    storeslot!(t::SlotTable, i::Int, x) = (t[i] = x; t)
+end
+
 """
     ClockCache{K, V}(; maxsize::Integer = 10_000, by = nothing)
 
@@ -8,37 +41,225 @@ never hit again are evicted first.
 
 `maxsize` limits the number of entries, or, when `by` is given, the sum of `by(value)` over
 all entries.
+
+Lookups take no lock and write no shared memory, so hits scale with the number of threads that
+share the cache. Insertions, evictions and the other changes take a lock. A lookup that races
+with a removal may still return the removed value, and one that races with an insertion may
+miss it; a returned value always belongs to its key.
 """
-mutable struct ClockCache{K, V} <: SlotCache{K, V}
-    const slots::Slots{V}
-    const ref::Vector{Bool}
-    const live::Vector{Bool}
-    hand::Int # last inspected slot
+mutable struct ClockCache{K, V} <: DictCache{K, V}
+    @atomic table::SlotTable
+    occupied::Int # slots of `table` holding an entry or a tombstone
+    const ring::Vector{Entry{K, V}} # the clock; free positions are unassigned
+    const freepositions::Vector{Int}
+    hand::Int # last inspected position of `ring`
+    count::Int
+    currentsize::Int
+    maxsize::Int
+    const by::Any
+    const counts::Vector{Int} # hits and misses per thread, see `countlookup!`
+    const lock::ReentrantLock
+
+    function ClockCache{K, V}(; maxsize::Integer = 10_000, by = nothing) where {K, V}
+        maxsize >= 0 || throw(ArgumentError("maxsize must be non-negative"))
+        counts = zeros(Int, 8 * nextpow(2, Threads.maxthreadid()))
+        return new{K, V}(emptytable(), 0, Entry{K, V}[], Int[], 0, 0, 0, maxsize, by, counts, ReentrantLock())
+    end
+end
+
+function emptytable(n::Int = 16)
+    t = SlotTable(undef, n)
+    foreach(i -> storeslot!(t, i, nothing), 1:n)
+    return t
 end
 
-ClockCache{K, V}(; maxsize::Integer = 10_000, by = nothing) where {K, V} =
-    ClockCache{K, V}(Slots{V}(maxsize, by), Bool[], Bool[], 0)
+# The slot after `i`, wrapping around (tables have a power-of-two length); a probe for `hash`
+# starts at `nextslot(t, hash % Int)`.
+nextslot(t::SlotTable, i::Int) = i & (length(t) - 1) + 1
 
-function admit!(c::ClockCache, i::Int)
-    i > length(c.ref) && (push!(c.ref, false); push!(c.live, false))
-    c.ref[i] = false
-    c.live[i] = true
+# The entry for the probe `p` (see `Key`), or `nothing`.
+findentry(::ClockCache, ::Nothing) = nothing
+@inline function findentry(c::ClockCache{K, V}, p::Key{P}) where {K, V, P}
+    t = @atomic :acquire c.table
+    i = nextslot(t, p.hash % Int)
+    for _ in 1:length(t) # bounds the probe also while slots change
+        e = loadslot(t, i)
+        e === nothing && return nothing
+        e isa Entry{K, V} && e.hash == p.hash && e.key isa P && isequal(p.key, e.key::P) && return e
+        i = nextslot(t, i)
+    end
+    return nothing
+end
+
+# Sets the reference bit only if it is clear, so hot entries are not written to.
+@inline function usevalue!(e::Entry)
+    (@atomic :monotonic e.used) || (@atomic :monotonic e.used = true)
+    return e.val
+end
+
+# Each thread counts in its own cache line (8 `Int`s), since a shared counter would bounce
+# between cores on every hit; a power of two of lines avoids a division. Tasks on one thread
+# cannot interleave between the load and the store, as neither yields.
+@enum LookupOutcome HIT = 1 MISS = 2
+@inline function countlookup!(c::ClockCache, outcome::LookupOutcome)
+    counts = c.counts
+    i = 8 * ((Threads.threadid() - 1) & (length(counts) ÷ 8 - 1)) + Int(outcome)
+    GC.@preserve counts begin
+        p = pointer(counts, i)
+        unsafe_store!(p, unsafe_load(p, :monotonic) + 1, :monotonic)
+    end
+    return nothing
+end
+
+function Base.get!(default::F, c::ClockCache{K, V}, key) where {F <: Base.Callable, K, V}
+    probe = _newkey(c, key)
+    e = findentry(c, probe)
+    if e !== nothing
+        countlookup!(c, HIT)
+        return usevalue!(e)
+    end
+    countlookup!(c, MISS)
+    v = convert(V, default())::V
+    sz = _entrysize(c, v)
+    @lock c.lock begin
+        # another task may have filled the key while we were computing
+        e = findentry(c, probe)
+        e === nothing || return e.val
+        addentry!(c, probe, v, sz)
+    end
+    return v
+end
+
+function Base.get(c::ClockCache, key, default)
+    e = findentry(c, _probe(c, key))
+    return e === nothing ? default : usevalue!(e)
+end
+Base.haskey(c::ClockCache, key) = findentry(c, _probe(c, key)) !== nothing
+Base.length(c::ClockCache) = @lock c.lock c.count
+Base.isempty(c::ClockCache) = length(c) == 0
+
+# The mutations below are called with the lock held.
+
+# Insert a key that is not present, evicting until it fits; a value larger than the whole cache
+# is not stored.
+function addentry!(c::ClockCache{K, V}, p::Key, v, sz::Int) where {K, V}
+    sz > c.maxsize && return c
+    while c.currentsize + sz > c.maxsize && c.count > 0
+        evict!(c)
+    end
+    if 2 * (c.occupied + 1) > length(c.table) # publish a new table without tombstones
+        t = emptytable(max(16, nextpow(2, 4 * (c.count + 1))))
+        for i in eachindex(c.ring)
+            isassigned(c.ring, i) && placeentry!(t, c.ring[i])
+        end
+        c.occupied = c.count
+        @atomic :release c.table = t
+    end
+    position = isempty(c.freepositions) ? length(resize!(c.ring, length(c.ring) + 1)) : pop!(c.freepositions)
+    e = c.ring[position] = Entry{K, V}(p.hash, p.key, v, false, sz, position)
+    placeentry!(c.table, e) === nothing && (c.occupied += 1)
+    c.count += 1
+    c.currentsize += sz
     return c
 end
 
-touch!(c::ClockCache, i::Int) = (c.ref[i] = true; c)
+# Store `e` in the first slot of its probe without an entry, and return what was there.
+function placeentry!(t::SlotTable, e::Entry)
+    i = nextslot(t, e.hash % Int)
+    while (old = loadslot(t, i)) isa Entry
+        i = nextslot(t, i)
+    end
+    storeslot!(t, i, e)
+    return old
+end
 
-forget!(c::ClockCache, i::Int) = (c.live[i] = false; c)
+function removeentry!(c::ClockCache, e::Entry)
+    t = c.table
+    i = nextslot(t, e.hash % Int)
+    while loadslot(t, i) !== e
+        i = nextslot(t, i)
+    end
+    if loadslot(t, nextslot(t, i)) === nothing
+        # no probe passes this slot or the tombstones before it, so they can all become empty
+        while true
+            storeslot!(t, i, nothing)
+            c.occupied -= 1
+            i = nextslot(t, i - 2)
+            loadslot(t, i) === TOMBSTONE || break
+        end
+    else
+        storeslot!(t, i, TOMBSTONE)
+    end
+    Base._unsetindex!(c.ring, e.position) # internal
+    push!(c.freepositions, e.position)
+    c.count -= 1
+    c.currentsize -= e.size
+    return c
+end
 
-# Terminates within two sweeps, since the first sweep clears every reference bit.
-function victim(c::ClockCache)
-    n = length(c.ref)
+# The hand clears set reference bits until it finds a clear one. Readers may set bits again
+# meanwhile, so after two sweeps the hand evicts the next entry anyway. Requires `c.count > 0`.
+function evict!(c::ClockCache)
+    n = length(c.ring)
+    steps = 0
     while true
         c.hand = c.hand >= n ? 1 : c.hand + 1
-        i = c.hand
-        c.live[i] || continue
-        c.ref[i] || return i
-        c.ref[i] = false
+        steps += 1
+        isassigned(c.ring, c.hand) || continue
+        e = c.ring[c.hand]
+        (steps > 2n || !(@atomic :monotonic e.used)) && return removeentry!(c, e)
+        @atomic :monotonic e.used = false
     end
     return
 end
+
+function Base.setindex!(c::ClockCache{K, V}, v, key) where {K, V}
+    probe = _newkey(c, key)
+    val = convert(V, v)::V
+    sz = _entrysize(c, val)
+    @lock c.lock begin
+        e = findentry(c, probe)
+        e === nothing || removeentry!(c, e)
+        addentry!(c, probe, val, sz)
+    end
+    return c
+end
+
+function Base.delete!(c::ClockCache, key)
+    @lock c.lock begin
+        e = findentry(c, _probe(c, key))
+        e === nothing || removeentry!(c, e)
+    end
+    return c
+end
+
+function Base.resize!(c::ClockCache; maxsize::Integer)
+    maxsize >= 0 || throw(ArgumentError("maxsize must be non-negative"))
+    @lock c.lock begin
+        c.maxsize = maxsize
+        while c.currentsize > c.maxsize && c.count > 0
+            evict!(c)
+        end
+    end
+    return c
+end
+
+function Base.empty!(c::ClockCache)
+    @lock c.lock begin
+        @atomic :release c.table = emptytable()
+        empty!(c.ring)
+        empty!(c.freepositions)
+        c.occupied = c.hand = c.count = c.currentsize = 0
+    end
+    return c
+end
+
+function cache_stats(c::ClockCache)
+    counts = c.counts
+    total(outcome) = GC.@preserve counts sum(i -> unsafe_load(pointer(counts, i), :monotonic), Int(outcome):8:length(counts))
+    return @lock c.lock (; hits = total(HIT), misses = total(MISS), length = c.count, c.currentsize, c.maxsize, c.by)
+end
+
+function snapshot(c::ClockCache{K, V}) where {K, V}
+    return @lock c.lock Pair{K, V}[c.ring[i].key => c.ring[i].val for i in eachindex(c.ring) if isassigned(c.ring, i)]
+end