ramify

A headless, config-driven uploader that fans one batch of files out to many FTP, FTPS, and SFTP endpoints concurrently. You point it at files and a list of endpoints; it uploads every file to every endpoint in parallel, verifies each transfer, retries transient failures, and exits non-zero if anything failed so a wrapping script can react. Run it from cron, CI, or a folder-watcher: one static binary, one YAML file.

$ ramify ./batch --config stocks.yml
[shutterstock] photo1.jpg: uploaded (verified: size) in 1.21s
[dreamstime] photo1.jpg: uploaded (verified: size) in 0.98s
[shutterstock] photo2.jpg: uploaded (verified: size) in 1.15s
[dreamstime] photo2.jpg: uploaded (verified: size) in 1.04s
[shutterstock] done: 2 succeeded, 0 failed in 3.10s
[dreamstime] done: 2 succeeded, 0 failed in 2.87s

Features

• One batch, many destinations. Declare endpoints in YAML; add or remove one without touching code. Every endpoint uploads in its own goroutine.
• Three protocols: ftp, ftps (explicit AUTH TLS), and sftp (password or private-key auth, host keys from ~/.ssh/known_hosts).
• Verification on by default. After each upload ramify compares the remote size against the local file to catch truncation. Disable with --no-verify.
• Retries with reconnect. A fixed number of attempts per file with a fixed delay, each attempt reconnecting from scratch. Configurable globally and per endpoint.
• Fail-fast preflight (--dry-run): connect, authenticate, and prove the target directory is writable with a self-deleting probe file, without touching the files you are sending.
• Automation-friendly output: quiet / default / verbose text, or newline-delimited JSON (--json), with meaningful exit codes and errors on stderr even under --quiet.
• Secrets via env vars: ${ENV_VAR} interpolation in any config field, resolved at load time. Literal values still allowed.
• Zero runtime dependencies: one static binary, no daemon.

Install

Homebrew (macOS / Linux)

brew install alexeyu/tap/ramify

go install

go install github.com/alexeyu/ramify/cmd/ramify@latest

Prebuilt binaries

Download the archive for your OS/architecture from the latest release, unpack it, and put ramify on your PATH. Linux, macOS, and Windows on amd64 and arm64 are built for every release; each archive is listed in checksums.txt.

Quick start

1. Write a config (stocks.yml) listing your endpoints. Start from config.yml.example, or see Configuration below for the full field reference.

2. Preflight it before you trust it to a cron job:

   $ export SHUTTERSTOCK_FTP_PASSWORD=...
   $ ramify ./batch --config stocks.yml --dry-run
   [shutterstock] dry-run ok: reachable and writable, would upload 12 files
   [dreamstime] dry-run ok: reachable and writable, would upload 12 files

3. Upload for real:

   $ ramify ./batch photo-extra.jpg --config stocks.yml

Positional arguments may be files or directories, mixed freely. Directories expand non-recursively: ramify takes every regular file and skips subdirectories and dotfiles. Two inputs sharing a basename would land under the same remote name, so ramify rejects them up front.

Configuration

Config is a YAML file with a list of endpoints plus optional global policy defaults. Any endpoint can override any policy key.

endpoints:
  - name: shutterstock          # unique label, used in output and errors
    protocol: ftps              # ftp | ftps | sftp
    host: ftp.shutterstock.com
    port: 21                    # optional; defaults to the protocol's standard port
    username: myuser
    password: ${SHUTTERSTOCK_FTP_PASSWORD}
    overwrite: delete-first     # delete-first (default) | direct

  - name: dreamstime
    protocol: sftp
    host: sftp.dreamstime.com
    username: myuser
    private_key: ~/.ssh/id_ed25519   # ~ is expanded

  - name: legacy-agency
    protocol: ftp
    host: ftp.legacy-agency.example
    username: myuser
    password: hunter2           # literal secrets allowed, not forced to env vars
    attempts: 5                 # per-endpoint override of the global default below

# Global policy defaults (each overridable per endpoint):
attempts: 3                     # total tries per file
retry_delay: 2s
connect_timeout: 30s            # bounds the whole connect + login, TCP dial included
stall_timeout: 5m               # fail a transfer idle this long; 0 disables
max_consecutive_connect_failures: 3   # write an endpoint off after this many connect failures in a row

Endpoint fields

Field	Applies to	Notes
name	all	Required, unique. Shown in all output.
protocol	all	ftp, ftps, or sftp. Required.
host	all	Required.
port	all	Optional; defaults to the protocol's standard port.
username	all	Required.
password	ftp, ftps, sftp	Required for ftp/ftps. For sftp, an alternative or complement to private_key.
private_key	sftp	Path to an SSH private key (~ expanded). Set both this and password and the key wins; password then serves as the passphrase for an encrypted key.
overwrite	all	delete-first (default) deletes any existing remote file first; direct uploads straight over it.
insecure_skip_verify	ftps	Disables TLS certificate verification. For self-signed / test servers only; never use against a production endpoint.

Policy fields (global or per-endpoint)

Field	Default	Meaning
attempts	3	Total tries per file before ramify counts it as failed.
retry_delay	2s	Fixed wait between attempts (Go duration string).
connect_timeout	30s	Bounds the entire connect + authenticate sequence.
stall_timeout	5m	Fails a transfer that makes no forward progress for this long. 0 disables.
max_consecutive_connect_failures	3	After this many connect failures in a row, ramify skips the endpoint's remaining files as unreachable instead of retrying each one.

Secrets

${ENV_VAR} interpolation works in any string field, resolved when the config loads. An unset variable is a load-time error, so you catch a typo before the run starts rather than as a confusing auth failure mid-run.

Literal secrets work but invite leaks. ramify warns when the config file is readable by group or others; chmod 600 stocks.yml and keep it out of version control.

Usage

ramify <path>... --config <file> [flags]

Flag	Effect
--config <file>	Path to the YAML config. Required.
--dry-run	Connect, authenticate, and probe each endpoint for writability, then report how many files would upload. No transfers, no deletes of real files.
--no-verify	Skip post-upload size verification.
--quiet	Suppress non-error stdout. Errors still print to stderr.
--verbose	Print the full event stream, including byte-level progress.
--json	Emit newline-delimited JSON instead of text (honors the verbosity level).
--version	Print version and exit.
--help	Print usage and exit.

--quiet and --verbose are mutually exclusive. Flags and paths may appear in any order. ramify treats everything after a bare -- as a path, so you can upload a file whose name starts with -:

$ ramify --config stocks.yml -- -weird-name.jpg

Output modes

Mode	stdout	stderr
--quiet	nothing	errors
default	per-file success + per-endpoint summary	errors
--verbose	full event stream incl. byte-level progress	errors

JSON output

--json reformats whatever the current verbosity level would print as one JSON object per line, keeping the same stdout/stderr split. Each line carries a type discriminator (file_start, progress, file_success, file_error, endpoint_unreachable, endpoint_given_up, endpoint_done, dry_run):

$ ramify ./batch --config stocks.yml --json
{"type":"file_success","endpoint":"shutterstock","file":"batch/photo1.jpg","verifyMethod":"size","durationSec":1.21}
{"type":"endpoint_done","endpoint":"shutterstock","succeeded":1,"failed":0,"durationSec":1.34}

Exit codes

Code	Meaning
0	Every file uploaded (and verified) on every endpoint.
1	Partial failure: at least one file failed after exhausting retries on at least one endpoint.
2	Configuration error (bad or invalid YAML, validation failures).
3	Usage error (bad flags, an input path that does not exist).

How it works

• Concurrency. One goroutine per endpoint. Within an endpoint, files upload sequentially over a single reused connection (connect once, upload all, disconnect), which amortizes the login cost and keeps ramify inside server limits on concurrent connections.
• Retries. A fixed attempt count with a fixed delay between tries. Each retry reconnects from scratch and re-runs verification, so a failed verification triggers a retry the same way a failed upload does.
• Verification. After each upload ramify compares the remote size against the local file (SIZE / Stat), which catches truncation cheaply on all three protocols. Hash verification waits for a later release, since SFTP has no standard equivalent. --no-verify turns it off.
• SSH host keys. ramify checks SFTP hosts against ~/.ssh/known_hosts and errors out on an untrusted host, with no insecure fallback, the same trust-on-first-use flow as OpenSSH. Record the key first by connecting once with ssh or ssh-keyscan.
• Cancellation. Ctrl-C / SIGTERM cancels cleanly: workers stop starting new transfers and retries, though an in-flight blocking transfer runs to completion first. Each endpoint still reports its results. A second Ctrl-C exits at once with code 130 rather than waiting out a stuck transfer.

Use as a Go library

The CLI is a thin wrapper over the ramify package. Upload returns a channel of typed events you range over:

package main

import (
	"context"
	"fmt"

	"github.com/alexeyu/ramify"
)

func main() {
	endpoints, err := ramify.LoadConfig("stocks.yml")
	if err != nil {
		panic(err)
	}

	files := []string{"photo1.jpg", "photo2.jpg"}
	events := ramify.Upload(context.Background(), files, endpoints, ramify.Options{})

	// You MUST keep receiving until the channel closes: workers send
	// unbuffered, so bailing out early would deadlock them.
	for ev := range events {
		switch e := ev.(type) {
		case ramify.FileSuccessEvent:
			fmt.Printf("%s -> %s ok\n", e.File, e.Endpoint)
		case ramify.FileErrorEvent:
			fmt.Printf("%s -> %s failed: %s\n", e.File, e.Endpoint, e.Reason)
		}
	}
}

You can also build []ramify.Endpoint yourself instead of loading YAML. See the package reference for the full event vocabulary and Options.

Building from source

Requires Go (see go.mod for the version). A Makefile wraps the common tasks:

make build              # build ./ramify with the version stamped in
make test               # unit tests, race detector on
make test-integration   # integration tests against real servers in Docker
make lint               # golangci-lint

Integration tests spin up real pure-ftpd and atmoz/sftp containers via the docker CLI, behind the integration build tag, so a plain go test ./... needs no Docker. Without a reachable Docker they skip with a message.

Releasing (maintainers)

GoReleaser cuts releases from a tag:

git tag v0.1.0
git push origin v0.1.0

The Release workflow then cross-compiles every target, attaches checksummed archives to the GitHub Release, and updates the Homebrew tap. Publishing the Homebrew cask requires a HOMEBREW_TAP_GITHUB_TOKEN repository secret: a token with write access to alexeyu/homebrew-tap. Without that secret the release still succeeds and skips only the cask push. Validate config changes locally with goreleaser check and dry-run a full build with goreleaser release --snapshot --clean.

Stability

Pre-1.0. The config schema and the Go library API may change between minor versions; v1.0.0 will freeze both.

License

MIT.