🦣 mammoth

big files. small api.

mammoth is a file storage typescript library. we use it in our web apps to store files like user uploads. it's content-addressed. files are streamed into a depot, identified by their blake3 hash, and deduplicated. mammoth works the same whether it's backed in memory, on disk, in the cloud, or in the browser's opfs. mammoth uses kv for bookkeeping.

npm install @e280/mammoth @e280/kv

import {Mammoth} from "@e280/mammoth"

const mammoth = new Mammoth()

🦣 mammoth stores files.

• write a file, and you get back its blake3 hash and size in bytes.

  const hash = await mammoth.write(blob.stream())

• read a file, identified by its hash.

  const blob = await mammoth.read(hash)

• delete a file.

  await mammoth.delete(hash)

🦣 mammoth is depot-agnostic.

• memory depot, for testing.

  import {Mammoth, MemoryDepot} from "@e280/mammoth"
  import {Kv} from "@e280/kv"
  
  const mammoth = new Mammoth(
  
    // depot is a bucket for blob storage
    new MemoryDepot(),
  
    // kv for metadata bookkeeping
    new Kv(),
  )

  defaults shown, this is equivalent:

  const mammoth = new Mammoth()

• disk depot, for nodejs servers, example using leveldb. (note the import paths)

  import {Mammoth, DiskDepot} from "@e280/mammoth/node"
  import {Kv, LevelMagazine} from "@e280/kv"
  import {Level} from "level"
  
  const mammoth = new Mammoth(
    new DiskDepot("./data/mammoth/depot"),
    new Kv(new LevelMagazine(new Level("./data/mammoth/kv"))),
  )

• opfs depot, with indexedDB kv for clientside in-browser storage.

  import {Mammoth, OpfsDepot} from "@e280/mammoth"
  import {Kv, idbOpen, IdbMagazine} from "@e280/kv"
  
  const mammoth = new Mammoth(
    new OpfsDepot(await navigator.storage.getDirectory()),
    new Kv(new IdbMagazine(await idbOpen("mammoth"))),
  )

🦣 more mammoth methods.

• check if a file exists, get back a boolean.

  const exists = await mammoth.has(hash)

• get file info, including size in bytes, added timestamp, and depot id.

  const {size, added, id} = await mammoth.info(hash)

• get stats, for the whole datalake.

  await mammoth.stats()
    // {count: 123, size: 123456789}

• loop over every file, with their hashes and infos.

  for await (const [hash, info] of mammoth.entries())
    console.log(hash, info)

🦣 mammoth-brained tips.

• analyze is for hashing files.

  import {analyze} from "@e280/mammoth"
  
  const {hash, size} = await analyze(blob.stream())

• analyze can help you avoid unnecessary uploads.

  const {hash} = await analyze(blob.stream())
  
  if (!await mammoth.has(hash))
    await mammoth.write(blob.stream())

• streamify makes a stream for a Uint8Array.

  import {streamify} from "@e280/mammoth"
  
  const readable = await streamify(new Uint8Array([0xDE, 0xAD, 0xBE, 0xEF]))

https://e280.org/