Skip to content
BonucciAndreaPublic

About

A small, fast array language that talks like q: tables, the whole join family, qSQL, tick data and dates, in one little C binary. Built on ngn/k.

Topics

Resources

Stars

40 stars

Watchers

1 watching

Forks

Latest commit

 

History

913 Commits

Folders and files

Repository files navigation

   █████╗ ███╗   ███╗██████╗ ███████╗██████╗
  ██╔══██╗████╗ ████║██╔══██╗██╔════╝██╔══██╗
  ███████║██╔████╔██║██████╔╝█████╗  ██████╔╝
  ██╔══██║██║╚██╔╝██║██╔══██╗██╔══╝  ██╔══██╗
  ██║  ██║██║ ╚═╝ ██║██████╔╝███████╗██║  ██║
  ╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚══════╝╚═╝  ╚═╝

A low-latency array language: columnar, vectorised, in-memory.

ci version license tests build

What is Amber

Amber is a small, fast, self-contained array language with the working vocabulary of q: dictionaries, tables & keyed tables with ([]…) literal syntax, the full join family (left · inner · union · plus · equi · as-of · window), qSQL-style select/by, strings, intraday tick / OHLC temporals, and column attributes implemented in C that turn search from O(n) into O(log n), which is ~1000–2000× faster on large data.

Amber's interpreter core is built on ngn/k, ngn's compact, AGPLv3 implementation of the K array language. Amber keeps that engine's speed and small footprint and layers a q vocabulary, C-level column attributes, native temporal types, ([]…) table syntax, a tick/HFT toolkit, and a modern REPL on top. (The attribution is recorded in NOTICE, as the AGPLv3 requires.)

t:([]sym:`AAPL`MSFT`AAPL; px:187.3 411.2 187.4; sz:100 250 50)   / a table, rendered instantly
qby[t; `sym; (,`vwap)!,{wavg[x`sz;x`px]}]                        / vwap by symbol

What's new in 2.9.0

q's other temporal types are in: minutes 09:30, seconds 09:30:15, months 2026.03m, timespans 0D01:00:00.000000000, and a typed null for each kind (0Nd to 0Nm). A list of dates, or of any one kind, is a real vector now, like q's, so sorting, grouping, finding and comparing them runs at int speed (10-25x faster; `date$ of timestamps 250x), and it prints and reads back the way q writes it (2026.01.01 0N, 2026.01 2026.02m, `date$()).

More of q's semantics too: x?y gives the count when y isn't there (it was 0N), int nulls stay null in arithmetic (0N+1 is 0N, +/ skips them as sum does), -3!x is q's show text, and an undefined name's error is the name. Against q itself, two batteries of everyday q lines are now 98.8% and 98.5% the same, and the temporal one 100%. What changed meaning is in the changelog.

And Amber talks to q now: q's wire format both ways (hopen, h x, the .z handlers, the timer, logs and -11!), so a kdb+ tickerplant runs on it as is. .q scripts read like q's, over several lines too.

2.8.0: q code runs like q. if do while, <> >= <=, q's casts (`long$x, "J"$s, `date$n, `year$d), ' to signal and @[f;x;h] to trap, upsert, `t insert, .Q.x and .j.x with their dot, and qSQL inside a lambda sees its arguments and locals. A run of about 2,000 everyday q lines against q itself went from 80% the same to 97%, and the rest are k choices Amber keeps on purpose or types it doesn't have yet (timespans, months). Repeated queries skip their parse, sorting symbols is 50x faster, and nothing got slower.

A few things changed meaning, to match q: type x gives q's numbers (@x still gives the letter), wj takes (q;aggs..), x^y of two lists the same length fills, and an error handler gets just the message ("type"). All in the changelog.

2.7.0 put tables on disk, the q way. `:db/t/ set t splays a table into binary column files that map straight into memory (a 10M-row table loads in ~6 ms), Q.dpft saves a partition, and \l db maps a whole date-partitioned database. select on it skips the partitions the where-clause rules out, reads only the columns it uses, and runs per partition when you group by date. Old text files still load.

2.7.1 makes qSQL a lot faster where it was slow (select distinct, like on symbols, update ... by, select by, the functional forms) and fixes update ... by ... where. 2.7.2 puts qSQL on all your cores (big where-clauses, maths, casts and symbol group-bys) and makes group-by on symbols 2-4x faster on one core too.

Attributes work like q now too: `s#x, `u#x, `p#x, `g#x, `#x and attr, on lists, tables and dicts, checked against q.exe case by case, including sorted dicts that look up by steps. Details in AMBER.md §9 and §9e.

2.6.0 before it brought threads and fusion: sort, find, in, distinct, group sums and float sums use every core once a vector is big, and float/int arithmetic inside a lambda runs as one fused pass. Details in the changelog.

Earlier releases

2.5.0, 2.4.2, 2.4.1, 2.4.0, 2.3.1, 2.3.0, 2.2.0 and everything before them are in CHANGELOG.md, which is the single place release notes live.

Quickstart

One folder, no install step, no system files. ./a builds if it needs to and drops you at the REPL; ./demo.sh builds and runs a 5,000,000-row tick session end to end.

git clone https://github.com/BonucciAndrea/amber && cd amber
./a                          # build if needed, then the REPL
AMBER_NATIVE=1 ./a           # -march=native build
./demo.sh                    # the full Mega Demo
./amber script.k             # run a script
./amber --help               # options and the \-command reference

A script gets the bare interpreter: put \l amber.k on its first line to load the standard library (sum, avg, xbar, the joins, qSQL's helpers). The REPL (./a) loads it for you.

\l amber.k
`0:$sum 1 2 3      / 6

Windows. WSL works as is, and so does plain Cygwin since 2.3. Install it with a compiler, make and git (python3 only if you'll run the test suite), either by ticking them in the installer or in one go:

setup-x86_64.exe -q -s https://mirrors.kernel.org/sourceware/cygwin/ -P gcc-core,make,git,python3

then open a Cygwin terminal and run the commands above. Clone with Cygwin's git (or set git config --global core.autocrlf false first) so the scripts keep their LF endings. The interpreter, the REPL and the test suite all run there; libamber.so is Linux and macOS only.

Typed at the prompt, where a bare table renders as a grid and qSQL works directly:

([]sym:`a`b`c; px:100 200 300)     / tables are first-class, no `show` needed

trade:([]sym:`a`b`a; time:3 4 9; px:100 200 300; sz:10 20 30)
quote:([]sym:`a`a`b`a; time:1 5 2 8; bid:10 11 20 12)
aj[`sym`time; trade; quote]        / as-of join, native C kernel

select last px by sym from trade   / qSQL, no sel"..." wrapper
select from trade where px>150
select[3;>px] from trade           / sorted and limited

`sym xasc trade                    / library dyads work infix
v:asc 2000000?1000000000           / `s attribute set by asc ...
v ? 12345 67890                    / ... so find is O(log n)

peach[{avg x?1.0}; 8#1000000]      / 8 tasks across real cores, no GIL

The demo, the full shell-rc recipe, a worked tour of every feature and the complete command list are in docs/QUICKSTART.md.

Terminal charts

Braille line charts, Unicode candlesticks and histograms, drawn in the terminal with no dependencies. Temporal axes are labelled as clocks and calendars rather than as the integers underneath:

gentq 1000                                          / sets `trades` and `quotes`

plot (14*{sin x%7}'!74;60;9)                        / braille line chart
tplot (trades`time; trades`px)                      / x axis reads 09:30:00
candle bars[10; select from trades where sym=`AAPL] / candlesticks, clock-labelled

Full gallery and the axis-unit rules: docs/QUICKSTART.md and docs/AMBER.md.

Native temporal types

Dates, times and timestamps are first-class types with literal syntax, auto-display and type-aware arithmetic, with no wrappers:

2026.07.30                          / date         -> 2026.07.30
10:00:00.000 + 00:00:05.000         / time + time  -> 10:00:05.000
2026.08.15 - 2026.07.30             / date - date  -> 16   (days)
2026.07.30D09:30:00.000000000       / timestamp (ns since 2000.01.01)
year 2026.07.30                     / accessors: year month day dow / thh tmm tss
"D"$"2026.12.25"                    / string casts: "D"$ "T"$ "P"$

The satellite ecosystem

Everything that consumes Amber lives outside this repository and reaches it through one of three seams: libamber.so, the in-process ext/ registry, or a TCP socket. Nothing below is mentioned anywhere in src/, and none of it is compiled, linked or configured by this build.

repository seam what it is
python-amber libamber.so pip install amber. Zero-copy NumPy views of Amber columns, pandas and Arrow bridges, dynamic dispatch (am.gentq(10_000_000)).
amber-arrow libamber.so ArrowArrayStream over an Amber table, batches that are windows onto one export, not slices of it. Plus amberd, the TCP query server, and an Arrow Flight daemon.
amber-jupyter python-amber A Jupyter kernel. Amber cells and %%python cells in one process, so a column crosses as a pointer.
vscode-amber amberd socket Syntax highlighting, and a standalone LSP daemon with qSQL-aware completion, idiom hovers and diagnostics that never evaluate.
grafana-amber-datasource amberd socket Live dashboards. Bare qSQL panels, column-oriented on the wire.
amber-flame python-amber / amberd A visual profiler, flamegraphs, Speedscope and Chrome tracing, built on the engine's own \trace.

The engine gained one build flag, one export map and one section of ext.h for all of it.

Why attributes matter

bench.k times 5,000 lookups with ? (find) on identical data, sorted-attributed vs not (2.4.1). Without the attribute find hashes the whole column every call; with it, it's a binary search per lookup:

rows plain sorted ( `s) speedup
100 k 3.2 ms 0.41 ms 7×
500 k 16.8 ms 0.94 ms 17×
2 M 184 ms 2.0 ms 93×
5 M 827 ms 2.9 ms 288×

Results are identical; only the time differs. asc / xasc set the attribute for you, and meta shows it in the a column.


Language notes (30-second version)

Amber uses a terse array notation. A few things worth knowing:

  • Two-argument library dyads work infix or in brackets. Since 2.0.0, x in y, t lj kt, `sym xasc t, 5 within 3 9, "/" sv `a`b`c all work infix, exactly like q, and the bracket form f[x;y] and prefix form f x still work. The infix set is a curated list (in within like lj ij uj aj aj0 wj wj1 pj ej cross inter union except ss sv vs xasc xdesc) plus the built-in symbol verbs (+ - * % ! & | < > = ~ , ^ # _ $ ? @ .). A name bound to a function of two arguments is infix too, as in q (f:{x+y}; 3 f 4), and a lambda written in place isn't (3 {x+y} 4).
  • <> >= <=, if do while work as in q (since 2.8.0), and type gives q's numbers (@x the letter).
  • qSQL is bare, at the prompt and in scripts. Type select … by … from … where … (also exec / update / delete) with no sel"…" wrapper; bare column names like wavg[sz;px] just work. Since 2.0.0 this bare form also works inside a .k file loaded once the stdlib is up (the loader runs each file through the same rewriter the REPL uses); the sel"…" / exq"…" / upd"…" / del"…" string forms and the functional forms qselect/qby/qwhere still work too.
  • peach[f;y] is real multi-core. It runs on a pool of AMBER_THREADS threads (default: the online CPU count; =1 forces serial), so heavy per-item work scales across cores with no GIL.
  • Big vectors use the threads on their own. Since 2.6, sort, grade, find, in, distinct, gather, group aggregates, float sums and fused float/int expressions split work over the same pool once a vector gets big (32K items and up). Small vectors stay on one thread, no overhead.
  • Grids preview Q-style. show t prints the first CROWS rows (default 20) then .., with a dimmed [N rows x M cols] footer and ANSI syntax highlighting.
  • Errors come as a Rust-style report: error[CODE], where it is, ^^^ under the token and a hint (see Rust-style diagnostics). AMBER_DIAG=0 gives the one-line caret form.
  • Symbols have no _. Use a quoted symbol `"a_b".
  • Tables: ([]col:vals;…); keyed tables: ([key:vals]col:vals). A bare table at the prompt auto-renders as a grid.

Full reference: docs/AMBER.md. Built-in help: \ lists every topic, then \q \j \z \m for the q layer, \0 \+ \` \' \: for the core, \o \i \g \r \e for disk, IPC, charts, speed and errors, \k for every REPL command and key, \v \ast \trace for the session/diagnostic tools (see REPL diagnostics), and \disasm for the bytecode disassembler (see Engine extensions).


Finance / HFT module (fin.k)

Auto-loaded after amber.k. Generate a market session and analyse it the way an HFT desk does:

gentq 100000                       / sets global `trades` and `quotes`
genopt 2000                        / sets global `options` (random option chain)
m:aj[`sym`time; trades; quotes]    / TAQ: prevailing quote for every trade
tsign m                            / Lee-Ready trade sign (+1 buy / -1 sell)
effspread m                        / effective spread = 2|px-mid|
qby[trades;`sym; `vwap!enlist {wavg[x`sz;x`px]}]   / VWAP by symbol
bars[1; trades]                    / 1-minute OHLCV bars

Included: mid spread spreadbps micro imbal (book), vwap twap tsign signedvol effspread notional (trades), ret logret rvol movavg movsum movmax movmin ema rollstd (returns/vol), bars symstats (aggregation), bysym symrows gidx (O(1) index), genopt gentq (generators). Walkthrough: ./amber examples/hft.k.

Attributes. Amber has all four of q's attributes in C: `s#x sorted, `u#x unique, `p#x parted, `g#x grouped (attr reads them, meta shows them; `sa and friends work too). Sorted int vectors stored 16 bits or wider give O(log n) kernel find; grouped + the group index give O(1) per-symbol slicing.

Documentation

docs/QUICKSTART.md the demo, the full install recipe, a worked tour, every command
docs/AMBER.md the language reference: every verb, adverb and qSQL clause
docs/REPL.md line editing, � / �st / race, error reports, grid modes
docs/INTERNALS.md architecture, extensions, SIMD/parallel, Arrow, libamber.so
docs/BENCHMARKS.md the full engine matrix, methodology, and the query files
docs/INSTALL.md shell configuration
docs/MISSING.md what is not implemented yet, and why
CHANGELOG.md every release

What's inside

file
a, build.sh launcher (build-if-stale) and portable compile (gcc / clang)
src/*.c, src/*.h the interpreter, ngn/k core + Amber extensions (src/p.c the ([]…) parser; src/ar.c Arrow; src/arena.{h,c} the HFT arena, 32-byte aligned; src/diagnostic.{h,c} the Rust-style formatter; the native aj kernel in src/a.c; src/inspect.{h,c} the \v inspector; src/ast.{h,c} the \ast visualiser; src/trace.{h,c} the \trace profiler; src/fmtutil.{h,c} and src/ansi.h shared formatting/colour helpers; src/simd.{h,c} AVX2/NEON/scalar kernels; src/parallel.{h,c} the thread pool and its parallel kernels; src/dsk.c the on-disk column files; src/vm.{h,c} the bytecode disassembler behind \disasm; src/csv.{h,c} the native CSV parser behind `csvr)
amber.k the q vocabulary (auto-loaded)
repl.k the REPL, banner, grid rendering, \grid/\clear, help; CRLF-safe module loader; reads its input through `rdl (the native editor) and exposes the optional ext.* hooks
src/ln.{h,c}, src/lnk.c the native line editor (raw termios, history) and the `rdl verb that the REPL reads through, this is what replaced rlwrap
src/ext.{h,c}, ext/ both extension seams. Sections 1-5: the in-process registry, runtime verbs plus \-command / editor / startup hooks, and the (empty by default) directory build.sh compiles out-of-tree extensions from. Section 6: the out-of-process dynamic C API behind libamber.so (amber_init, amber_eval_str, amber_get_vector_ptr, …)
src/libamber.map the linker export map for the shared build: only amber_* and am_ext_* reach a host process's dynamic namespace, so the engine's terse internals (mr, run, add, …) cannot collide with a host's symbols
fin.k finance / HFT module (auto-loaded), see \m help
std.k qsql.k temporal.k sys.k hdb.k ipc.k tick.k modules (auto-loaded)
examples/ tour.k · basics.k · tick.k · hft.k · peach.k · wj.k · graphs.k · …
test.k test-fin.k test-ext.k assertion suites (202 + 35 + 79)
tests/harness.k shared assertion harness, t (value), tv (trapped expression), te (must-raise), tk (must-not-raise), hexpect (assertion-count guard), hreport
tests/test_matrix.k 309-case combinatorial matrix: every primitive × every element type × sizes 0 / 1 / 10 / 100 000+ (crossing the SIMD and PAR_THRESHOLD boundaries), asserted as invariants (shape, algebraic identity, vector-kernel-vs-scalar-reference) rather than frozen literals
tests/test_qsql.k 117-case qSQL matrix, written in the bare select … from t syntax you type (run through the same qrw rewrite the REPL and loader apply): the full select/exec/update/delete clause lattice, multi-key by, empty / single-row / heavily-duplicated tables, and malformed queries asserted to raise cleanly
tests/fuzz.py malformed-input & deep-nesting crash fuzzer, asserts a clean K error, never a signal or a hang
tests/test_capi.{c,sh} the dynamic C API: 81 assertions against libamber.so, linked as a satellite would link it (src/ext.h and nothing else from src/), run once at -O2 and once under ASan + UBSan with leak detection, the ownership rules in the header are prose, and this is what checks them
tests/test_qsql_script.sh, tests/test_comments.sh shell suites for the 2.0.0 loader work: bare qSQL inside a loaded .k file, and the unterminated bare-/ comment now raising cleanly
tests/run_tests.sh runs all of the above (--asan re-runs everything under ASan + UBSan)
tests/test_repl_term.py pty-driven REPL terminal suite: asserts no rlwrap: diagnostic ever reaches a session, that termios is byte-for-byte restored after a normal exit and after ^C, that the editing keys really edit, and that piped/non-tty behaviour is unchanged
tests/test_ext_seam.sh, tests/ext_probe.c installs a miniature extension into ext/, checks the verb / \-command / --help hooks fire and that the engine's own suite is unaffected, then uninstalls it and checks the engine is back to stock
tests/*.c standalone C test harnesses: test_simd.c/test_parallel.c (no Amber dependency), test_ast.c (links the full interpreter, ast.c is inherently built on Amber's real parser)
bench.k bench-fin.k bench-std.k bench/ attribute / index / window benchmarks; bench/run_comparative.py cross-engine harness (see docs/BENCHMARKS.md §5); bench/queries/amber_*.k and bench/queries/k_*.k are separate, independently-tuned scripts per engine, see Comparative benchmark query files
docs/ AMBER.md (reference) · MISSING.md (roadmap + known leniencies) · BENCHMARKS.md
CHANGELOG.md release history (2.0.0 first)
.gitattributes forces LF checkout of sources so the REPL's line-based loader works on Windows too

Roadmap

Amber covers a large slice of q. docs/MISSING.md is an honest map of what's next; top picks: wiring the `g grouped attribute into the C find path; map-reduce for queries that span partitions; the missing atom types (short/real/byte/guid); and live REPL syntax highlighting (colouring tokens as you type, which needs repl.k's raw-keystroke input loop rewritten).

Isolation

Amber is a single self-contained folder. The interpreter is named amber (never k or q), built only inside the folder, never placed on your PATH. It reads/writes no config, no QHOME, no dotfiles. Your other k and q installs are untouched; deleting the folder uninstalls Amber completely.

Licence

GNU AGPLv3 (see LICENSE). Amber's interpreter core derives from ngn/k, an AGPLv3 K interpreter by ngn; that attribution is preserved in NOTICE, as the licence requires. Amber is an independent language and is not affiliated with, nor a distribution of, that project.

About

A small, fast array language that talks like q: tables, the whole join family, qSQL, tick data and dates, in one little C binary. Built on ngn/k.

Topics

Resources

Stars

40 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages