Freeing an array from another task while its owner is capturing puts the free in the graph #1123

unsafe_free! enqueues the free on the stream that last used the memory, whichever task calls it. If the task that owns that stream is capturing a graph at that moment, the hipFreeAsync becomes part of the capture. The resulting graph has a MemFree node for memory the graph never allocated, and launching it crashes:

using AMDGPU
capturing, freed = Base.Event(), Base.Event()
bref = Ref{Any}()
t = Threads.@spawn begin
    b = AMDGPU.zeros(Float32, 1 << 20); AMDGPU.synchronize()
    bref[] = b
    c = AMDGPU.zeros(Float32, 16); AMDGPU.synchronize()
    AMDGPU.capture() do
        notify(capturing); wait(freed)
        c .+= 1f0
    end
end
wait(capturing)
AMDGPU.unsafe_free!(bref[])  # enqueued on t's stream, which is capturing
notify(freed)
g = fetch(t)
AMDGPU.HIP.launch(AMDGPU.HIP.instantiate(g))

julia: .../clr/hipamd/src/hip_graph_internal.hpp:2714: virtual void hip::GraphMemFreeNode::VirtualMemFreeNode::submit(amd::device::VirtualDevice&): Assertion `phys_mem_obj != nullptr' failed.

This happens on f0d0502 with ROCm 7.2.4 on a gfx1036 iGPU, using julia -t 8. @gbaraldi ran into the same kind of graph on MI300A in #1120, with a HIP build without assertions, and there hipGraphLaunch segfaulted. In that case, recycling had handed the stream to the capturing task. Synchronizing on such an array from another task probably has the same problem: synchronize(::Managed) queries the capturing stream, or launches a host function on it.

Finalizers don't hit this, because capture disables the GC. Explicit frees and syncs from other tasks do. One way to fix it would be to check whether the memory's stream is capturing before using it from another task, and then defer the free until the capture is done.