Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/npu.md
Original file line number Diff line number Diff line change
Expand Up @@ -1135,6 +1135,15 @@ working directory's files are not the importer's to clean. The 6 hours are what
shared directory, so the number stays above the longest import; it is not the knob for disk or tmpfs pressure. The test suite sets one per run and
removes it at exit.

The cache directory is shared: every import of the same pack on the box uses the same file name. The copy is therefore
written under a temporary name beside it (`1bit-fconst19tmp-<pid>-<n>-<name>`) and renamed into place, so an import finds
either no copy or a complete one (engine#1985). Before that the copy was written in place, and two imports of one pack
started a fraction of a second apart could leave the first reading a file the second had just truncated. The result was a
wrong layout reading with exit 0: `Qwen3-1.7B` reported 280 refused tensors, `qk_norm` false and 0 measured tiles in 4 of
24 runs of three imports 0.15 s apart, and 0 of 144 with the rename. A temporary file left by a killed run is removed by
the 6-hour rule above. A binary older than this change still writes in place, so a box where old and new binaries import
the same pack at the same moment can still show it.

### Linking the kernel set the container will be served with

A model directory carries its own NPU kernels in `npu/` (see "A model directory"), and for an imported
Expand Down
51 changes: 45 additions & 6 deletions npu/pack_import.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <nlohmann/json.hpp>

#include <algorithm>
#include <atomic>
#include <cmath>
#include <memory>
#include <functional>
Expand All @@ -32,6 +33,7 @@
#include <map>
#include <set>
#include <stdexcept>
#include <unistd.h>

namespace onebit::npu::pack_import {
namespace {
Expand Down Expand Up @@ -782,12 +784,8 @@ std::string canonical_token_fconst(const std::string& path, int hidden, int head
}
out.insert(out.end(), fc.map + tail_off, fc.map + fc.size);
unmap_file(fc.map, fc.size);
{
std::ofstream os(work, std::ios::binary | std::ios::trunc);
if (!os) throw std::runtime_error(work + ": cannot write the canonical copy of the 1.9 .fconst");
os.write(reinterpret_cast<const char*>(out.data()), std::streamsize(out.size()));
if (!os) throw std::runtime_error(work + ": short write of the canonical copy of the 1.9 .fconst");
}
// engine#1985: never in place. The path is shared by every import of this pack on the box, see write_canonical_copy().
write_canonical_copy(work, out.data(), out.size());
return work;
}

Expand Down Expand Up @@ -6133,6 +6131,47 @@ int run(int argc, char** argv) {
}


// engine#1985: the temporary name a canonical copy is written under before it is renamed onto `work`. Same directory
// (rename is only atomic inside one filesystem). The name starts with the scratch stem `1bit-fconst19`, so the age rule
// of evict_stale_canonical_copies() removes one a killed run left behind, and continues with `tmp-`, so it can never
// match a pack's `1bit-fconst19v<n>-...` prefix: another import's sweep must not delete a copy that is being written.
std::string canonical_copy_temp_name(const std::string& work) {
static std::atomic<unsigned> serial{0};
const fs::path p(work);
const std::string base = "1bit-fconst19tmp-" + std::to_string(static_cast<long>(::getpid())) + "-" +
std::to_string(serial.fetch_add(1)) + "-" + p.filename().string();
return (p.parent_path() / base).string();
}

// engine#1985: write the canonical copy under a private name and rename it onto `work`. The copy used to be written in
// place with O_TRUNC, at a path every import of the same pack shares, and reuse is decided by length -- so a second
// import that started while the first was still writing saw a short file, truncated it and began again, and the first
// then read its own copy while it was being refilled. Measured on Qwen3-1.7B with three imports 0.15 s apart: 4 of 24
// runs read "0 in-block offsets", refused all 280 attention tensors and exited 0, on a pack a lone import reads with 0
// gaps. With a rename a reader finds no copy or a complete one, and two writers each install the same bytes.
void write_canonical_copy(const std::string& work, const uint8_t* data, size_t len) {
const std::string tmp = canonical_copy_temp_name(work);
{
std::ofstream os(tmp, std::ios::binary | std::ios::trunc);
if (!os) throw std::runtime_error(tmp + ": cannot write the canonical copy of the 1.9 .fconst");
os.write(reinterpret_cast<const char*>(data), std::streamsize(len));
os.flush();
if (!os) {
os.close();
std::error_code rm;
fs::remove(tmp, rm);
throw std::runtime_error(tmp + ": short write of the canonical copy of the 1.9 .fconst");
}
}
std::error_code ec;
fs::rename(tmp, work, ec);
if (ec) {
std::error_code rm;
fs::remove(tmp, rm);
throw std::runtime_error(work + ": cannot move the canonical copy of the 1.9 .fconst into place (" + ec.message() + ")");
}
}

// Public (engine#1662): see the declaration in pack_import.h. Lives outside the anonymous namespace because the test
// calls it directly with crafted files, so the decision that guards a ~291 MB rewrite is pinned without a 1.9 pack.
bool canonical_copy_reusable(const std::string& work, int64_t expected_len) {
Expand Down
10 changes: 10 additions & 0 deletions npu/pack_import.h
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,16 @@ Plan plan(const std::string& pack_dir);
// and a directory on the scratch path must all come back false.
bool canonical_copy_reusable(const std::string& work, int64_t expected_len);

// engine#1985: how the canonical copy reaches `work`. It is written under canonical_copy_temp_name(work) -- a private name
// in the same directory -- and renamed into place, so a concurrent import of the same pack finds no copy or a complete one,
// never one that is being written. Writing in place let a second import truncate the file the first was about to read.
// Throws std::runtime_error when the copy cannot be written or moved; the temporary file is removed on either failure.
// Exposed so the rule has an arm that needs no pack (threads writing one path while a reader checks every complete-length
// file it finds), and so the temporary name's two properties are pinned: a pack's sweep prefix never matches it, and the
// age rule does.
std::string canonical_copy_temp_name(const std::string& work);
void write_canonical_copy(const std::string& work, const uint8_t* data, size_t len);

// Removes the canonical Token .fconst copies in `dir` that nothing will reuse (engine#1770). Reuse writes a new copy when the
// source changes rather than overwriting, so without this the scratch directory grows one multi-hundred-MB file per change; on
// this box 699 files / 8.7 GiB of tmpfs. Three rules:
Expand Down
99 changes: 99 additions & 0 deletions tests/npu_pack_import_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@
#include <unistd.h>

#include <algorithm>
#include <thread>
#include <atomic>
#include <cmath>
#include <cstdio>
#include <cstring>
Expand Down Expand Up @@ -836,6 +838,102 @@ void test_external_import_success(const std::string& root) {
// "<tensor>.qweight.empty: no data", which reports a placeholder as though the bytes were absent rather than elsewhere.
// Crafted, so this needs no 1.9 pack: a graph with one resolvable tensor, one MatMulNBits whose qweight input is a
// `.empty` placeholder with no sibling, and a model.bin next to it.
// engine#1985: the canonical copy lives at a path every import of the same pack shares, and it used to be written in place.
// A second import that started mid-write saw a short file, truncated it and wrote again, and the first then read its own
// copy while it was being refilled: Qwen3-1.7B came back with 280 refused tensors and exit 0 in 4 of 24 runs of three
// imports 0.15 s apart. The arms: the bytes arrive and no temporary file stays; a failed write leaves nothing; the
// temporary name is out of reach of another import's prefix sweep and inside the age sweep; and the race itself, with
// threads doing what an import does (reuse the copy if it is complete, else write it, then read it).
void test_write_canonical_copy() {
const fs::path dir = fs::temp_directory_path() / ("1bit-atomic-" + std::to_string(::getpid()));
std::error_code ec;
fs::remove_all(dir, ec);
fs::create_directories(dir);
const std::string prefix = "1bit-fconst19v1-7-2048-16-28-";
const std::string name = prefix + "957724736-1234.fconst";
const std::string work = (dir / name).string();
const size_t n = size_t(4) << 20;
std::vector<uint8_t> body(n);
for (size_t i = 0; i < n; ++i) body[i] = uint8_t((i * 2654435761u) >> 24) | 1; // never 0: a refilled hole reads as 0
auto same = [&](const std::string& f) {
std::ifstream is(f, std::ios::binary);
if (!is) return false;
std::vector<uint8_t> got(n + 1);
is.read(reinterpret_cast<char*>(got.data()), std::streamsize(got.size()));
return size_t(is.gcount()) == n && std::memcmp(got.data(), body.data(), n) == 0;
};
auto entries = [&] {
int k = 0;
for (const auto& e : fs::directory_iterator(dir)) { (void)e; ++k; }
return k;
};

pi::write_canonical_copy(work, body.data(), n);
check(same(work), "atomic copy: the bytes written are the bytes at the path");
check(entries() == 1, "atomic copy: no temporary file is left beside the copy");
std::vector<uint8_t> other(body);
other[n / 2] ^= 0xff;
pi::write_canonical_copy(work, other.data(), n);
check(!same(work) && entries() == 1, "atomic copy: a second write replaces the copy, again leaving one file");
pi::write_canonical_copy(work, body.data(), n);

bool threw = false;
try {
pi::write_canonical_copy((dir / "no-such-dir" / name).string(), body.data(), n);
} catch (const std::runtime_error&) { threw = true; }
check(threw && entries() == 1, "atomic copy: a copy that cannot be written throws and leaves nothing behind");

// the temporary name: beside the copy, not the copy, unique per call, invisible to the prefix sweep, visible to the age sweep
const std::string t1 = pi::canonical_copy_temp_name(work), t2 = pi::canonical_copy_temp_name(work);
const std::string b1 = fs::path(t1).filename().string();
check(fs::path(t1).parent_path() == fs::path(work).parent_path(), "atomic copy: the temporary file is in the copy's directory");
check(t1 != work && t1 != t2, "atomic copy: the temporary name is not the copy's and is new on every call");
check(b1.compare(0, prefix.size(), prefix) != 0 && b1.compare(0, 13, "1bit-fconst19") == 0,
"atomic copy: the temporary name starts with the scratch stem and not with the pack's prefix");
{ std::ofstream o(t1, std::ios::binary); o << "being written"; }
pi::evict_stale_canonical_copies(dir.string(), name, prefix, std::string(), 3600);
check(fs::exists(t1) && fs::exists(work), "atomic copy: another import's sweep does not remove a copy that is being written");
fs::last_write_time(t1, fs::file_time_type::clock::now() - std::chrono::hours(2));
pi::evict_stale_canonical_copies(dir.string(), name, prefix, std::string(), 3600);
check(!fs::exists(t1) && fs::exists(work), "atomic copy: a temporary file a killed run left is removed by the age rule");

// the race: every thread is an import. It reuses a complete-length copy or writes one, then reads the path, as
// canonical_token_fconst()'s caller does. One thread also removes the copy now and then, so the cycle restarts.
fs::remove(work, ec);
std::atomic<int> bad{0}, read_ok{0}, wrote{0};
auto importer = [&](int id) {
for (int r = 0; r < 24; ++r) {
if (!pi::canonical_copy_reusable(work, int64_t(n))) {
try { pi::write_canonical_copy(work, body.data(), n); ++wrote; } catch (const std::runtime_error&) { ++bad; }
}
std::error_code e2;
// An open that fails is the remover having been there, not a bad copy; a copy that OPENS must be whole.
if (std::ifstream(work, std::ios::binary).good()) {
std::ifstream is(work, std::ios::binary);
std::vector<uint8_t> got(n + 1);
if (is) {
is.read(reinterpret_cast<char*>(got.data()), std::streamsize(got.size()));
if (size_t(is.gcount()) == n && std::memcmp(got.data(), body.data(), n) == 0) ++read_ok;
else ++bad;
}
}
if (id == 0 && r % 4 == 3) fs::remove(work, e2);
}
};
std::vector<std::thread> pool;
for (int i = 0; i < 6; ++i) pool.emplace_back(importer, i);
for (auto& t : pool) t.join();
check(bad.load() == 0, "atomic copy: no import ever reads a copy another import is still writing (engine#1985)");
check(read_ok.load() > 0 && wrote.load() > 1, "atomic copy: the race arm really wrote more than once and read copies back");
int leftover = 0;
for (const auto& e : fs::directory_iterator(dir))
if (e.path().filename().string() != name) ++leftover;
check(leftover == 0, "atomic copy: the race leaves no temporary files");
fs::remove_all(dir, ec);
std::printf(" atomic canonical copy: 11 arms (bytes, replace, failed write, temporary name x4, %d racing writes / %d reads, no leftovers)\n",
wrote.load(), read_ok.load());
}

// engine#1662: the ~291 MB canonical Token .fconst rewrite is skipped when the copy on the scratch path already holds
// exactly the bytes this run would write. The decision is a length test against a length that is a function of the source
// alone, so what can go wrong is the test being too willing -- a truncated copy from an interrupted run, a leftover
Expand Down Expand Up @@ -5534,6 +5632,7 @@ int main(int argc, char** argv) {
test_external_container_names();
test_external_packed_genre(golden_dir.empty() ? "." : golden_dir);
test_external_import_success(golden_dir.empty() ? "." : golden_dir);
test_write_canonical_copy();
test_canonical_copy_reusable(); test_evict_stale_canonical_copies();
test_sweep_fconst_scratch_dirs();
test_sweep_is_wired(golden_dir.empty() ? "." : golden_dir);
Expand Down
Loading