Modular TypeScript media extraction and download toolkit. Each site integration is split into a provider, parser, transformer, and pipeline so extraction rules, output mapping, and download planning stay easy to update independently.
Requires Node.js 22.12 or newer.
npm install downfluxpnpm add downfluxExtraction only. This is the default and writes nothing but a JSON record of what was found:
import { BeegProvider } from 'downflux/providers';
const provider = new BeegProvider('https://beeg.com/example-video-url');
const result = await provider.getVideo();
console.log(result);Downloading to disk. Provider methods resolve as soon as extraction finishes, so you learn what is
coming without waiting for it; whenSettled() waits for the transfers:
import { BeegProvider } from 'downflux/providers';
import { OutputType } from 'downflux/types';
const provider = new BeegProvider('https://beeg.com/example-video-url')
.setOutput(OutputType.DEVICE, { directoryPath: '/srv/media' });
const metadata = await provider.getVideo();
const { downloaded, failed, errors } = await provider.whenSettled();whenSettled() resolves even when some items fail, because a partial batch is a normal result -
inspect failed and errors. It rejects only if the download pipeline itself could not run.
Install it globally and the command is on your path:
npm install -g downfluxdownflux <url> -o ./media -q 1080pAs a project dependency it is linked into node_modules/.bin, so reach it through your
package manager or a script:
npx downflux <url> -o ./media{
"scripts": {
"fetch": "downflux"
}
}To try it without installing anything:
npx --yes downflux@latest <url>USAGE
downflux <url> [options]
downflux providers [filter]
OPTIONS
-o, --output <dir> directory to download into (default: ./DownFlux)
-q, --quality <q> preferred video quality
-m, --method <name> provider method to call (default: the first one it exposes)
-c, --concurrency <n> parallel downloads (default: 5)
--json print extracted metadata, download nothing
--no-progress suppress the progress panel
-h, --help show this help
-v, --version show the version
The URL decides the provider: its host is matched against providerPatterns, and a host no
provider claims falls back to generic extraction with a warning. Providers do not share one
entry point — most expose getVideo, some only getVideos or getMetadata — so the CLI
calls the first method a provider actually has, or whichever you name with --method.
--json prints the extracted metadata and downloads nothing, so it pipes into jq. Ctrl+C
during a download aborts it, removes the partial file and exits 130.
There is no root entry. Every import names the package directory it comes from, so a symbol has exactly one import path and you only pay for what you name.
import { OutputType } from 'downflux/types';
import { FileManager } from 'downflux/storage';
import { BeegProvider } from 'downflux/providers';| Subpath | Contents |
|---|---|
downflux/providers |
every site integration |
downflux/base |
BaseProvider, BaseParser, BaseTransformer, BasePipeline, BaseStrategy |
downflux/core |
coordinators, registries, exceptions, downfluxTheme |
downflux/engines |
HttpClient (page and API requests; downloads go through charanko) |
downflux/storage |
FileManager (JSON output and resource types, on top of charanko's) |
downflux/shared |
constants, helpers |
downflux/types |
enums and type-level helpers |
downflux/contracts |
interfaces only, erased at runtime |
Bundled cost of a single subpath, measured against a real install:
| Import | Bundle |
|---|---|
downflux/providers |
269.8 KB |
downflux/storage |
13.0 KB |
downflux/types |
4.9 KB |
DownFlux uses ffmpeg when HLS or fragmented media needs to be finalized into a playable file. The package includes ffmpeg-static, but some package managers can block its postinstall script.
If you use pnpm in the consuming app, approve the bundled binary build:
pnpm approve-buildsSelect ffmpeg-static, then reinstall dependencies if needed.
You can also install ffmpeg yourself:
brew install ffmpegThen point DownFlux at that executable:
import { BeegProvider } from 'downflux/providers';
import { OutputType } from 'downflux/types';
await new BeegProvider('https://beeg.com/example-video-url')
.setOutput(OutputType.DEVICE, { directoryPath: 'downloads' })
.setJobOptions({
transcodeOptions: {
ffmpegPath: '/opt/homebrew/bin/ffmpeg'
}
})
.getVideo();setOutput() decides what a job produces. Downloading modes run their transfers in the background,
so the bytes arrive through pipelineHooks.onDownload and completion comes from whenSettled().
| Mode | Downloads | How you receive it | Memory | Use for |
|---|---|---|---|---|
JSON (default) |
no | Writes a .json metadata file, returns metadata |
trivial | Cataloguing, debugging a provider |
RETURN |
no | Returns metadata, writes nothing | trivial | Feeding your own pipeline or database |
DEVICE |
yes | Files on disk under dirConfig.directoryPath |
constant | Anything large; batch jobs; media libraries |
STREAM |
yes | A Readable per item, via onDownload |
constant | HTTP responses, object storage uploads |
DEVICE streams to a .part file and renames on success, so an interrupted run never leaves a
truncated file at the real name. Pass an absolute directoryPath on a server; relative paths
resolve against process.cwd(). A URI such as s3://bucket/media is rejected - that is a remote
target, not a filesystem path, so use STREAM and pipe to your storage client instead.
import { PornHubProvider } from 'downflux/providers';
import { OutputType } from 'downflux/types';
import { pipeline } from 'stream/promises';
await new PornHubProvider(url)
.setOutput(OutputType.STREAM)
.setJobOptions({
pipelineHooks: [
{
onDownload: async ({ result }) => {
reply.writeHead(200, {
'content-type': result.mimeType,
// present only when bytes pass through untouched
...(result.sizeBytes ? { 'content-length': String(result.sizeBytes) } : {}),
// omit this header to play inline instead of downloading
'content-disposition': `attachment; filename="${result.originalFilename}"`
});
await pipeline(result.stream!, reply);
}
}
]
})
.getVideo();Consume the stream promptly; an unread one stalls the transfer behind the pipe buffer.
Segmented sources are remuxed on the fly into fragmented MP4, which a pipe can carry but a
regular MP4 cannot. Those responses have no Content-Length, so the browser shows no percentage and
cannot seek. result.estimatedBytes is a best-effort source size for progress UI - never send it as
Content-Length. When seeking matters, download with DEVICE and serve the finished file.
import { Upload } from '@aws-sdk/lib-storage';
onDownload: async ({ result }) => {
await new Upload({
client: s3,
params: { Bucket: 'my-bucket', Key: result.originalFilename, Body: result.stream, ContentType: result.mimeType }
}).done();
}DEVICE jobs handle SIGINT/SIGTERM by default: in-flight transfers abort, each deletes its own
.part file, the job reports ABORTED, and the process exits once cleanup finishes. A second Ctrl+C
exits immediately.
Other modes leave signals alone, since servers own their own shutdown. Set abortOnSignal to
override either default, or drive cancellation yourself:
const controller = new AbortController();
const provider = new PornHubProvider(url)
.setOutput(OutputType.DEVICE, { directoryPath: '/srv/media' })
.setJobOptions({ signal: controller.signal, abortOnSignal: false });Call provider.dispose() when finished to detach the progress renderer, or
dispose({ closeConnections: true }) to also close the shared undici pools.
setProgressLogging(true) renders a live panel that redraws in place, with a bar per concurrent
item, transfer rate and ETA. It is drawn on stderr by inumaki,
and becomes one summary line per update when stderr is not a TTY or the logger is in append mode.
It looks the same from the downflux command and inside your app: DOWNFLUX banner and branding, live
panel. If your app calls Inumaki.create({ theme, mode }) first, the panel follows that instead.
provider.setProgressLogging(true, { captureConsole: true });captureConsole routes your own console.log around the live region so it is not erased by the next
frame. It is off by default because it patches stdout/stderr, which is the host application's
call to make.
For programmatic progress use onProgress, which receives inumaki's ProgressSnapshot on every
update: status, counts (Targets, Items), details, totals (bytes, totalBytes, speed, eta)
and transfers, one per file with its own bytes, segments, rate and ETA.
The generated Markdown API docs live in docs-md.
Useful entry points:
- BaseProvider - public provider API and fluent execution options.
- BaseParser - shared HTML extraction helpers.
- BaseTransformer - fetches pages and normalizes parser output.
- BasePipeline - turns extracted metadata into downloadable items.
- ExecutionCoordinator - job-level extraction and output flow.
- TaskCoordinator - concurrency, hooks, and background downloads.
- HttpClient - page and API requests with provider host fallbacks.
- FileManager - JSON results and resource types.
- charanko - the download engine: HTTP and HLS transfers, CDN fallbacks, refreshes, sinks, and FFmpeg remux.
- JobSettlement - what
whenSettled()hands back;onProgressreceives inumaki'sProgressSnapshot. - Provider, OutputType, ExecutionType, VideoQuality.
Regenerate the Markdown docs with:
pnpm run docs:mdflowchart TD
User([Caller chooses Provider])
subgraph ProviderLayer["Provider Layer"]
ServiceProvider[ServiceProvider]
BaseProvider[BaseProvider<br/>• Validates URL<br/>• Builds execution request]
end
subgraph TransformLayer["Transformation Layer"]
TransformerRegistry[TransformerRegistry<br/>Resolves provider transformer]
Transformer[Transformer<br/>Coordinates extraction lifecycle]
HttpClient[HttpClient<br/>Fetches HTML / APIs]
ParserRegistry[ParserRegistry<br/>Loads matching parsers]
Parser[Parser<br/>• Extracts metadata<br/>• Detects media sources<br/>• Finds identifiers]
end
subgraph PipelineLayer["Pipeline Layer"]
PipelineRegistry[PipelineRegistry<br/>Loads provider pipeline]
Pipeline[Pipeline<br/>Builds execution items]
PipelineItems[Pipeline Items]
URLs[URLs<br/>Media / Playlist URLs]
MediaTypes[Media Types<br/>Video / Audio / Images]
Identifiers[Identifiers<br/>Content IDs / Metadata]
end
subgraph ExecutionLayer["Execution Layer"]
TaskCoordinator[TaskCoordinator<br/>Schedules concurrent tasks]
Charanko[Charanko<br/>Downloads one item]
end
subgraph StreamLayer["Transfer & Output Layer (charanko)"]
StreamHttpClient[StreamHttpClient<br/>Streams remote files<br/>• Uses Strategy for CDN fallback<br/>• Delegates HLS to HlsClient]
StrategyRegistry[StrategyRegistry<br/>Loads extraction strategies]
Strategy[Strategy<br/>• Defines extraction rules<br/>• Provider-specific behavior]
HlsClient[HlsClient<br/>Processes m3u8 playlists]
FileManager[FileManager<br/>Handles sinks & temp files]
FFmpegEngine[FFmpegEngine<br/>Muxing / Conversion / Finalization]
Output[Output<br/>JSON / Device File / Stream]
end
User --> ServiceProvider
ServiceProvider --> BaseProvider
BaseProvider --> TransformerRegistry
TransformerRegistry --> Transformer
Transformer --> HttpClient
HttpClient --> ParserRegistry
ParserRegistry --> Parser
Parser --> Transformer
Transformer --> PipelineRegistry
PipelineRegistry --> Pipeline
Pipeline --> PipelineItems
PipelineItems --> URLs
PipelineItems --> MediaTypes
PipelineItems --> Identifiers
Pipeline --> TaskCoordinator
TaskCoordinator --> Charanko
Charanko --> StreamHttpClient
StreamHttpClient --> StrategyRegistry
StrategyRegistry --> Strategy
Strategy -.->|CDN Fallback<br/>Re-extraction| StreamHttpClient
StreamHttpClient --> HlsClient
StreamHttpClient --> FileManager
HlsClient --> FileManager
FileManager --> FFmpegEngine
FileManager --> Output
FFmpegEngine --> Output
classDef provider fill:#4f46e5,color:#fff,stroke:#312e81,stroke-width:2px;
classDef transform fill:#0f766e,color:#fff,stroke:#134e4a,stroke-width:2px;
classDef pipeline fill:#ca8a04,color:#fff,stroke:#854d0e,stroke-width:2px;
classDef execution fill:#dc2626,color:#fff,stroke:#7f1d1d,stroke-width:2px;
classDef network fill:#0369a1,color:#fff,stroke:#0c4a6e,stroke-width:2px;
classDef parser fill:#7c3aed,color:#fff,stroke:#581c87,stroke-width:2px;
classDef strategy fill:#8b5cf6,color:#fff,stroke:#6d28d9,stroke-width:2px;
classDef stream fill:#059669,color:#fff,stroke:#064e3b,stroke-width:2px;
classDef output fill:#ea580c,color:#fff,stroke:#7c2d12,stroke-width:2px;
class ServiceProvider,BaseProvider provider;
class TransformerRegistry,Transformer transform;
class PipelineRegistry,Pipeline,PipelineItems,URLs,MediaTypes,Identifiers pipeline;
class TaskCoordinator,Charanko execution;
class HttpClient network;
class ParserRegistry,Parser parser;
class StrategyRegistry,Strategy strategy;
class StreamHttpClient,HlsClient stream;
class FileManager,FFmpegEngine stream;
class Output output;
In short: a provider creates a typed request, coordinators run the extraction/download flow, registries load the correct provider-specific classes, engines handle network transport, pipelines decide what should be downloaded, and storage writes or returns the result.
| Site | Provider | Short description |
|---|---|---|
| AnalRz new | AnalRzProvider |
MP4 downloads; under development. |
| PimpBunny | PimpBunnyProvider |
MP4 downloads, HLS downloads; geo-sensitive, under development. |
| Beeg | BeegProvider |
MP4 downloads, HLS downloads; geo-sensitive, under development. |
| BlackPorn new | BlackPornProvider |
MP4 downloads; under development. |
| BoKepPorn new | BoKepPornProvider |
MP4 downloads, KVS metadata; under development. |
| ColliderPorn | ColliderPornProvider |
MP4 downloads, HLS downloads, embeds; geo-sensitive, under development. |
| CumLouder | CumLouderProvider |
MP4 discovery; under development. |
| DaFreePorn new | DaFreePornProvider |
MP4 downloads, KVS metadata; under development. |
| DaNude new | DaNudeProvider |
MP4 downloads, KVS metadata; under development. |
| EpicGfs | EpicGfsProvider |
MP4 downloads, KVS metadata; under development. |
| EPorner | EPornerProvider |
MP4 downloads, HLS downloads; external API, geo-sensitive, under development. |
| HqPorn | HqPornProvider |
MP4 downloads; under development. |
| Interracial new | InterracialProvider |
MP4 downloads, KVS metadata; under development. |
| ItsPorn new | ItsPornProvider |
MP4 downloads, KVS metadata; under development. |
| Lesbian8 | Lesbian8Provider |
MP4 downloads, KVS metadata; under development. |
| MegaTube | MegaTubeProvider |
MP4 downloads, KVS metadata; under development. |
| MomVids new | MomVidsProvider |
MP4 downloads, KVS metadata; under development. |
| MyLust | MyLustProvider |
MP4 downloads; under development. |
| OkPorn | OkPornProvider |
HLS downloads; under development. |
| PerfectGirls | PerfectGirlsProvider |
HLS downloads; under development. |
| Porn300 | Porn300Provider |
MP4 discovery; under development. |
| PornDoe | PornDoeProvider |
MP4 downloads; external API, under development. |
| PornHub | PornHubProvider |
MP4 downloads, HLS downloads, KVS metadata; under development. |
| PornId | PornIdProvider |
MP4 downloads, KVS metadata; under development. |
| PornOne | PornOneProvider |
MP4 downloads; Cloudflare challenge, under development. |
| PornSeven | PornSevenProvider |
MP4 discovery; under development. |
| PornsOk | PornsOkProvider |
MP4 downloads; under development. |
| PussySpace | PussySpaceProvider |
MP4 downloads; external API, under development. |
| SexVid | SexVidProvider |
MP4 downloads, KVS metadata; under development. |
| Shameless | ShamelessProvider |
MP4 downloads, KVS metadata; under development. |
| SuperPorn | SuperPornProvider |
MP4 downloads; under development. |
| SxyPorn | SxyPornProvider |
MP4 downloads; Cloudflare challenge, under development. |
| TheyAreHuge | TheyAreHugeProvider |
MP4 downloads, KVS metadata; login required, under development. |
| TnAFlix | TnAFlixProvider |
MP4 downloads; geo-sensitive, under development. |
| TubeVSex | TubeVSexProvider |
MP4 downloads; under development. |
| WallHaven | WallHavenProvider |
Provider-specific extraction; under development. |
| XCafe | XCafeProvider |
MP4 downloads; under development. |
| XDegu | XDeguProvider |
MP4 downloads, KVS metadata; under development. |
| XGroovy | XGroovyProvider |
MP4 downloads; under development. |
| XHamster | XHamsterProvider |
MP4 downloads, HLS downloads; under development. |
| XnXX | XnXXProvider |
MP4 downloads, HLS downloads; under development. |
| Xozilla new | XozillaProvider |
MP4 downloads, KVS metadata; under development. |
| XVideos | XVideosProvider |
MP4 discovery, HLS downloads, KVS metadata; under development. |
| ZbPorn new | ZbPornProvider |
MP4 downloads, KVS metadata; under development. |
| ZzzTube | ZzzTubeProvider |
MP4 downloads; under development. |
These are scaffolding. They are exported and registered like any other provider, but throw
NotImplementedException rather than emitting placeholder output, so an unfinished provider fails
loudly instead of producing files that look valid. Providers whose metadata declares
canDownload: false or requiresBrowser: true reject downloads up front for the same reason.
| Site | Provider | Short description |
|---|---|---|
| ArtStation new | ArtStationProvider |
Art gallery extraction; under development. |
| Behance new | BehanceProvider |
Portfolio platform extraction; requires API, under development. |
| Bluesky new | BlueskyProvider |
Social media extraction; requires API, under development. |
| Danbooru new | DanbooruProvider |
Anime image board extraction; under development. |
| DeviantArt new | DeviantArtProvider |
Art community extraction; requires API, under development. |
| Flickr new | FlickrProvider |
Photo sharing extraction; under development. |
| Gelbooru new | GelbooruProvider |
Anime image board extraction; under development. |
| Imgur new | ImgurProvider |
Image hosting extraction; under development. |
| Instagram new | InstagramProvider |
Social media extraction; login required, Cloudflare challenge, under development. |
| MangaDex new | MangaDexProvider |
Manga reader extraction; under development. |
| Mastodon new | MastodonProvider |
Social media extraction; geo-sensitive, under development. |
| Newgrounds new | NewgroundsProvider |
Art portal extraction; under development. |
| Pexels new | PexelsProvider |
Stock photo extraction; under development. |
| Pinterest new | PinterestProvider |
Image sharing extraction; requires API, Cloudflare challenge, under development. |
| Pixiv new | PixivProvider |
Art community extraction; requires API, login required, under development. |
| Reddit new | RedditProvider |
Social media extraction; under development. |
| TikTok new | TikTokProvider |
Video platform extraction; requires API, Cloudflare challenge, under development. |
| Tumblr new | TumblrProvider |
Social media extraction; under development. |
| Twitter new | TwitterProvider |
Social media extraction; requires API, login required, Cloudflare challenge, under development. |
| Unsplash new | UnsplashProvider |
Stock photo extraction; under development. |
| WikiArt new | WikiArtProvider |
Art museum extraction; under development. |
| Wikimedia new | WikimediaProvider |
Media library extraction; geo-sensitive, under development. |
More incoming...
Requires Node.js 22.12 or newer.
pnpm install
pnpm run build
pnpm run docs:md.gitignore ignores *.md, so newly generated documentation pages need forcing:
git add -f docs-mdTo add a new provider you can run this following commands.
Use the --no-[registryCoordinatorType] to exclude that file.
Available flags: strategy | transformer | pipeline | method
pnpm run make:registry [registryName] OR
pnpm run make:registry --no-strategy TwitterAfter running the previous command, run this commands sequentially for barrel imports and auto entries
pnpm run make:index
pnpm run format
pnpm run buildThe generated docs are intentionally linked from this README so provider and API documentation remain browsable from the repository root.
Three separate commands, in order. They are separate on purpose: publishing to npm is irreversible — a version number can never be reused — so it is never reached as the tail of a longer chain that might get there after something earlier went wrong, and either distribution step can be retried without redoing the other.
# 1. version, land and tag — no distribution happens here
pnpm run release # or release:major for a breaking release
# 2. publish the package
pnpm run publish:npm
# 3. attach the tarball to a GitHub release on the tag
pnpm run publish:githubStep 1 bumps the version, writes the CHANGELOG, opens and squash-merges the release pull
request, then tags the merged commit. It refuses to start unless you are on main, the
working tree is clean, and main is level with origin/main. Two ordering rules are baked
in, and are why this is a script rather than a chain of git commands:
- the release branch is cut before the version bump, because
standard-versioncommits to the current branch; bumping onmainleaves a commit that the squash merge duplicates, andmaindiverges from its own remote - the tag is created after the merge, against the squashed commit; tagging earlier strands the tag on a commit that never reaches the released history
If step 1 fails, nothing further is attempted. Fix the cause and re-run, or tag by hand with
pnpm run release:tag, which refuses to tag anything that is not already on origin/main.
pnpm run release:dry-run previews the bump and changelog without writing anything, and
pnpm run pack:dry-run lists the files npm will receive.