Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ChubDL - Character Hub Downloader

A command-line tool to download character cards, gallery images, lorebooks and character JSON files from characterhub.org / chub.ai.

Features

  • Download all character cards from a user, from a list of users, or a single card by URL (both the author/slug and the numeric id style)
  • Download gallery images, lorebooks and the raw character JSON
  • Version tracking: detects updated cards from the API listing alone (no re-downloading), keeps every version of a card's data, and stores the picture only once
  • Re-downloads files that went missing, skips everything that is current
  • Embeds taglines and gallery descriptions into image metadata (lossless)
  • Saves the card's current version, not the stale copy chub serves as an image
  • Configurable random delay between downloads to stay polite to the site
  • Automatic retries with exponential backoff, and Retry-After is honoured
  • Interrupted downloads never leave half-written files behind
  • Ctrl-C is safe: progress is saved after every download
  • Windows-safe filenames by default, Unicode and emoji preserved

Installation

git clone <this repo>
cd ChubDL
pip install -r requirements.txt

The web interface additionally needs fastapi and uvicorn (included in requirements.txt). Nothing else is required for the command line.

exiftool is optional. It is not a Python package, so it cannot be installed from requirements.txt:

  • Linux: sudo apt install libimage-exiftool-perl / sudo pacman -S perl-image-exiftool
  • macOS: brew install exiftool
  • Windows: download from https://exiftool.org and put exiftool.exe on your PATH

Without it the built-in python metadata writer is used (see below).

Usage

python chub_dl.py USERNAME [options]     command line
python chub_dl.py --ui                   web interface

Put USERNAME first. -x and -t accept an optional method name, so a username placed right after them would be read as the method.

Single cards

python chub_dl.py https://chub.ai/characters/author/some-card-a1b2c3d4
python chub_dl.py author/some-card-a1b2c3d4
python chub_dl.py -c author/card-one -c https://characterhub.org/lorebooks/author/some-lorebook

A positional argument containing a / is treated as a card path. Cards land in downloads/AUTHOR/ alongside a normal user download and share its version manifest, so a later full download of that author skips them. Lorebook URLs are downloaded as lorebooks. -g, -j, -t and -x apply as usual.

Chub serves every card under two URLs, and both are accepted:

python chub_dl.py https://chub.ai/characters/author/some-card-a1b2c3d4
python chub_dl.py https://chub.ai/characters/1234567
python chub_dl.py characters/1234567
python chub_dl.py -c characters/1234567 -c lorebooks/54321

An id says nothing about who owns the card, so id references are looked up before anything is downloaded and the owner is read from the response. The card then lands in that author's folder and shares their version manifest exactly as the author/slug form does, so the same card given either way is recognised as one card and downloaded once.

The space in front of an id (characters/ or lorebooks/) is only a hint: the API ignores it and answers with the real one, so lorebooks/1234567 still downloads character 1234567 as a character.

A bare number is read as a username, not an id, because numeric usernames exist and would otherwise be unreachable. Write characters/1234567 or the full URL to mean the card.

Web interface

python chub_dl.py --ui

Starts a local server on http://127.0.0.1:8765/ and opens it in your browser. The settings panel on the left applies to everything you start. Three tabs:

  • Jobs: paste usernames or card URLs (either style) one per line and press Start. Jobs run one at a time in the background with live progress bars and log output, and can be cancelled.

  • Browse: type a username to see their cards with thumbnails, taglines and a status badge (new, on disk, current, update, missing). Tick cards, or use "Select new + updates", and download only those. "Track" adds a card or the whole user to the tracked list.

  • Tracked: the contents of the tracked file (tracked_users.txt by default), users and single cards, with add/remove. "Check all tracked for updates" runs everything in one job.

  • History: every card ChubDL has downloaded, newest first, with whether each one is tracked. Filter by name or user, or tick "Only cards that are not tracked" to find cards you grabbed once and never set up for updates, then track them individually or several at a time. A card tracked because its whole user is tracked says so, and cannot be untracked on its own.

    The history is read from the version manifests that already exist in each user folder, so nothing extra is recorded to produce it and removing a download from the list is not currently possible - the manifest is also what makes update detection work.

The form remembers its last values in chubdl_ui_settings.json. --port, --host, --no-browser and --batch-file adjust the server. The web UI only listens on localhost. It always uses the default API host; --api-host is a command line option only.

Batch mode

python chub_dl.py USERNAME -B        # add USERNAME to tracked_users.txt and exit
python chub_dl.py -P -g -b -j        # process every user in tracked_users.txt
python chub_dl.py -P -f my_list.txt  # use a different list

The tracked file has one entry per line: a username, or a single card as author/slug, characters/ID or a chub.ai URL. Entries are normalised to their short form when added, so the same card added by either URL style is stored (and tracked) once. Blank lines and lines starting with # are ignored. Entries are processed in file order. -B accepts card references too, so python chub_dl.py author/slug -B tracks one card.

Options

General

  • -o, --output-dir DIR - Output directory (default: ./downloads)
  • -d, --delay SECONDS - Delay between downloads. A number or a range like 1-3 (default: 1-2)
  • -r, --max-retries N - Retries for failed requests, with backoff (default: 3)
  • -a, --api-host HOST - API host (default: sub.chub.ai)
  • -s, --no-sanitizing - Keep Windows-incompatible characters in filenames
  • -v, --verbose - Show debug output on the console
  • --log-dir DIR - Where to write log files (default: logs)
  • --no-log-file - Do not write a log file

What to download

  • -g, --download-gallery - Download gallery images
  • -l, --gallery-limit N - Maximum gallery images per character (default: 50)
  • -G, --force-gallery-check - Check galleries even when the card itself was skipped
  • -b, --download-lorebooks - Download lorebooks
  • -j, --download-character-json - Download character JSON files
  • -p, --download-presets - Presets (not implemented, only reports how many exist)

Metadata

  • --no-tagline-in-notes - Do not put the tagline at the top of the card's creator notes
  • -t, --add-taglines [METHOD] - Also embed the tagline into the card's image metadata
  • -x, --add-descriptions [METHOD] - Embed gallery descriptions into the images
  • -n, --no-fallback - Do not fall back to the next method when one fails
  • --no-refresh-cards - Keep the card data chub embedded at first upload (see below)

Repair

  • --repair-cards - One-off fix for cards downloaded before the above existed

Version tracking

  • --no-version-tracking - Disable tracking, only skip files that already exist
  • -J, --track-json-versions - Also track versions of character JSON files

Examples

python chub_dl.py SomeAuthor
python chub_dl.py SomeAuthor -g -l 10
python chub_dl.py SomeAuthor -g -x -t
python chub_dl.py SomeAuthor -g -x sidecar -n
python chub_dl.py SomeAuthor -o ./my_characters -d 2-4
python chub_dl.py SomeAuthor -b -j -J
python chub_dl.py SomeAuthor -v --no-log-file

Version tracking

The API listing includes a lastActivityAt timestamp for every project. Each user folder has a manifest, downloads/USERNAME/USERNAME_versions.json, that remembers which timestamp each downloaded file corresponds to, so on the next run the tool knows what changed before downloading anything. The manifest travels with the folder. If an old single-file version_tracking.json from earlier versions of this tool is present in the working directory, a user's section is imported from it the first time that user is processed.

One image is kept per card. An update to a card nearly always means new card data rather than new artwork, so an update does not copy the whole image:

  • First download: Name (id).png, plus the definition filed in Name (id) Versions/
  • Updated later: the new definition is filed as another version and written into the image already on disk. Nothing is re-downloaded but a few KB, and there is no second copy of the picture
  • If the artwork itself has changed - which does happen - the new image is saved as Name (id) [YYYY-MM-DD].png and becomes the current one, with the previous picture kept
  • If a tracked file is deleted, it is downloaded again
  • Gallery images are remembered by URL, so a reordered gallery only downloads the images that are actually new

Whether the artwork changed is settled with a HEAD request comparing the image's ETag, length and modification date against what was stored when it was last fetched, so the check itself costs nothing. Cards downloaded before this existed have no stored fingerprint; the first update after that keeps the image on disk and records a fingerprint for next time.

Versions

Every definition a card passes through is kept in Name (id) Versions/, named for the edit it came from:

Character1 (12345) Versions/
├── 2025-05-11 12-31-04.json
└── 2026-08-11 20-50-00.json

Versions are a few KB each next to a multi-megabyte image, which is what makes keeping all of them practical. The newest is what gets written into the PNG, so the image always carries the current card while the folder holds the history - and any older version can be put back into a PNG later without downloading anything, since the picture and the data are stored separately.

Character JSON files are only tracked with -J; otherwise they are skipped when the file exists. Lorebooks are always skipped when the file exists.

With --no-version-tracking every file is simply skipped when it already exists and updates are never detected.

Tagline in creator notes

Chub stores a card's page description in the card's creator_notes field. By default the tagline is added above it, in both the downloaded PNG and the JSON export, so it shows up wherever creator notes are displayed:

---
The tagline of the card
---

The description chub already put here...

The image itself is not re-encoded; only the embedded card data changes. Turn it off with --no-tagline-in-notes.

Note that a card whose creator notes open with its own <style> scaffolding may render differently, since the tagline block is inserted ahead of it.

Card versions and what chub actually serves

The image behind a card, chara_card_v2.png, is written once when the card is first uploaded and is never rewritten afterwards. Editing a card - adding a greeting, changing creator notes - updates the card's repository but leaves that image untouched, so its embedded data stays at version one forever. This is not true of every card: older cards do have images matching a later edit, so chub appears to have stopped regenerating them at some point.

The current definition lives only in the card's repository, which is also where -j has always taken its JSON from. So after downloading a card image, ChubDL fetches that definition and writes it into the PNG, replacing the stale copy:

  • Only the embedded text chunk changes. The image data is copied through byte for byte, so the picture is untouched and the file does not grow when this is repeated.
  • It happens before the tagline step, so the tagline is added to the current creator notes rather than to an old copy of them.
  • --no-refresh-cards turns it off and keeps the card exactly as chub serves it, stale data included.

Repairing cards downloaded earlier

Cards downloaded before this existed hold the first-upload data while the manifest recorded the API's edit time against them, so they look current and would never be corrected on their own. Those entries are recognisable: they have no definition field.

Fixing them is a one-off you trigger, not a check that runs forever. The web UI shows a banner on the Jobs tab while any card needs it, with a button that repairs them; on the command line it is --repair-cards:

python chub_dl.py --repair-cards

Either way only the definition is fetched and re-embedded, a few KB per card - no image is downloaded again. It reads the download manifests and nothing else, so anything you put in the output folder yourself is left alone. An entry whose file has since been deleted is skipped and reported; it sorts itself out the next time that card is downloaded.

If a card also carries a ccv3 chunk it is left alone, since the repository definition is a v2 card; a warning is logged in that case.

Metadata methods

-t and -x take an optional method. With no method, auto tries them in this order:

  • exiftool - Handles PNG, JPEG and WEBP. Needs exiftool installed.
  • python - Built in. Writes a PNG text chunk or JPEG EXIF (needs piexif) without re-encoding the image, so the card's embedded data and the image bytes are untouched. Cannot write WEBP.
  • sidecar - Writes a .txt file with the same name next to the image.

pillow is accepted as an alias for python. Taglines are stored as Description, gallery descriptions as UserComment. Text is stored as UTF-8 / UTF-16, so non-English descriptions survive.

Filenames

By default names are made Windows-safe: < > : " | ? * and control characters become -, whitespace is collapsed, trailing dots and spaces are trimmed, and reserved device names such as CON get a leading underscore. Emoji and other Unicode are kept. -s turns this off. Path separators are always removed regardless, because names come from the API and must never be able to write outside the output directory. Very long names are trimmed to fit the filesystem limit.

Directory structure

downloads/
└── USERNAME/
    ├── USERNAME_users_api_response.json    (latest raw API response)
    ├── USERNAME_versions.json              (version manifest)
    ├── Character1 (12345).png              (current image + current data)
    ├── Character1 (12345) [2026-02-03].png (kept only if the artwork changed)
    ├── Character2 (12346).png
    ├── Character1 (12345) Versions/
    │   ├── 2025-05-11 12-31-04.json        (every definition, oldest first)
    │   └── 2026-08-11 20-50-00.json
    ├── Character1 (12345) Gallery/
    │   ├── Character1 (12345)_project_api_response.json
    │   ├── Character1 1.webp
    │   └── Character1 2.png
    ├── JSONs/
    │   └── Character1 (12345).json
    └── Lorebooks/
        └── Lorebook1 (222).json

Project layout

chub_dl.py          launcher (CLI or --ui)
chubdl/config.py    Config dataclass shared by both frontends
chubdl/core.py      downloading, version tracking, metadata
chubdl/cli.py       command line frontend
chubdl/web.py       FastAPI backend for the web interface
chubdl/static/      the single-page web UI

Notes

  • Placeholder card images (the server's 200x200 / 240x240 "missing" image) are detected and discarded rather than saved. The old archive fallback host is offline, so there is currently no second source for those cards.
  • Every run writes a log to logs/ unless --no-log-file is given.
  • --api-host applies to the user listing, card lookup and gallery endpoints. JSON exports always come from api.chub.ai. api.chub.ai, sub.chub.ai and gateway.chub.ai return identical data as far as this tool can tell.

License

MIT

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages