diff --git a/di/grafana/grafana.md b/di/grafana/grafana.md new file mode 100644 index 00000000..ccf06823 --- /dev/null +++ b/di/grafana/grafana.md @@ -0,0 +1,147 @@ +# `grafana.q` – Grafana JSON datasource adaptor for kdb-x + +A library that lets a [Grafana](https://grafana.com/) server query a kdb+ +process directly, using the +[SimPod JSON datasource](https://github.com/simPod/GrafanaJsonDatasource) API. + +Once initialised, the module installs HTTP handlers on the process so that +requests carrying the `X-Grafana-Org-Id` header are intercepted and answered +with Grafana-shaped JSON, while all other HTTP requests fall through to any +existing handler. Point a SimPod JSON datasource at the process's HTTP port and +its tables become available as Grafana panels. + +--- + +## :sparkles: Features + +- Serves the `/search` endpoint – populates Grafana's metric dropdowns from the + tables in the process. +- Serves the `/query` endpoint – returns either **timeseries** or **table** + panel data. +- Answers the Grafana test-connection `GET` with a `200 OK`. +- Wraps any pre-existing `.z.pp`/`.z.ph` handlers rather than overwriting them. +- No connection to other modules required – loads standalone with an injected + logger. + +--- + +## :inbox_tray: Import + +```q +q)grafana:use`di.grafana +``` + +Only `init` is exported; everything else runs automatically via the installed +HTTP handlers. + +--- + +## :electric_plug: Dependencies + +| Dependency | Required | Description | +|---|---|---| +| `log` | yes | A dictionary of `` `info`warn`error `` logging functions, each with the `{[c;m]}` signature (context symbol; message string). | + +The logger is **mandatory** – `init` errors immediately if it is not supplied. +A ready-made logger can be taken from `di.log`, or you can pass your own: + +```q +/ custom logger +mylog:`info`warn`error!( + {[c;m] -1 "INFO [",string[c],"] ",m;}; + {[c;m] -1 "WARN [",string[c],"] ",m;}; + {[c;m] -2 "ERROR [",string[c],"] ",m;}); +``` + +--- + +## :gear: Configuration + +Configuration is optional and passed to `init` under the `config` key. Any +omitted value keeps its default. + +| Key | Type | Default | Description | +|---|---|---|---| +| `timecol` | symbol | `` `time `` | Name of the time column used for timeseries queries. | +| `sym` | symbol | `` `sym `` | Name of the column used to split data by instrument. | +| `timebackdate` | timespan | `2D` | How far back to look when finding distinct syms for the dropdowns. | +| `ticks` | long | `1000` | Number of rows returned for a table request. | +| `del` | char | `"."` | Delimiter separating the arguments within a query target. | + +--- + +## :memo: Initialisation + +`init` takes a single dictionary of dependencies (and optional config), then +installs the HTTP handlers. The handlers are installed only once, so `init` may +be called again to update the logger or configuration without re-wrapping. + +```q +/ logger only, defaults for everything else +q)grafana.init[enlist[`log]!enlist mylog] + +/ logger plus configuration overrides +q)grafana.init[`log`config!(mylog;`timecol`ticks!(`ts;500))] +``` + +--- + +## :wrench: Exported functions + +### `init` + +``` +init[deps] +``` + +- `deps` – a dictionary, one of: + - `` enlist[`log]!enlist logdict `` – inject the logger, use config defaults. + - `` `log`config!(logdict;configdict) `` – inject the logger and override config. +- `logdict` – `` `info`warn`error `` ! three `{[c;m]}` functions (required). +- `configdict` – any subset of the keys in the [configuration](#gear-configuration) table. + +Errors if no logger is supplied. Returns nothing; its effect is wiring the +logger, applying configuration, and installing the `.z.pp`/`.z.ph` handlers. + +--- + +## :mag: Query target syntax + +The Grafana metric strings produced by `/search` encode the table, panel type +and arguments, separated by `del` (default `"."`): + +| Prefix | Meaning | +|---|---| +| `t.` | table panel for the whole table | +| `t.
.` | table panel filtered to one sym | +| `g.
` | graph panel, one series per numeric column | +| `g.
.` | graph panel, one series per sym for a column | +| `o.
...` | "other" panels (single-stat, gauge, etc.) | +| `f.<...>` | the target is a function call rather than a table | + +--- + +## :rocket: Example usage + +```q +q)grafana:use`di.grafana +q)log:use`di.log / or define your own logger +q)grafana.init[enlist[`log]!enlist `info`warn`error!(log.info;log.warn;log.error)] + +q)trade:([]time:.z.p-0D00:00:01*til 100;sym:100#`a`b`c;price:100?100f) +``` + +Add a SimPod JSON datasource in Grafana pointing at this process's HTTP port, +then build panels using the metrics offered in the dropdowns (`t.trade`, +`g.trade.price`, …). + +--- + +## :test_tube: Tests + +Tests are written for k4unit and run with: + +```q +q)k4unit:use`di.k4unit +q)k4unit.moduletest`di.grafana +``` diff --git a/di/grafana/grafana.q b/di/grafana/grafana.q new file mode 100644 index 00000000..c960d5ee --- /dev/null +++ b/di/grafana/grafana.q @@ -0,0 +1,259 @@ +/ grafana json datasource adaptor for kdb-x +/ implements the simpod json datasource api so a grafana server can query +/ kdb+ tables and timeseries data over http - the /search endpoint populates +/ the grafana dropdowns and the /query endpoint returns timeseries or table data + +/ ----------------------------------------------------------------------------- +/ configuration - defaults applied here, overridden via the config dict in init +/ ----------------------------------------------------------------------------- + +/ name of the time column used for timeseries queries +timecol:`time; +/ name of the sym column used to split data by instrument +sym:`sym; +/ how far back to look when finding distinct syms +timebackdate:2D; +/ number of ticks to return for table requests +ticks:1000; +/ delimiter separating the arguments within a query target +del:"."; + +/ json type for each kdb datatype, keyed by .Q.t character +types:.Q.t!`array`boolean,(3#`null),(5#`number),11#`string; +/ milliseconds between 1970.01.01 and 2000.01.01 +epoch:946684800000; + +/ ----------------------------------------------------------------------------- +/ request handling +/ ----------------------------------------------------------------------------- + +zpp:{[x] + / parse a grafana http post request and dispatch to the matching handler + / cut at the first whitespace to isolate the api url from any function params + r:(0;n?" ")cut n:first x; + rqt:.j.k r 1; + .z.m.loginfo[`grafana;"received ",(r 0)," request"]; + handler:$["query"~r 0;query;"search"~r 0;search;annotation]; + :.[handler;enlist rqt;{[e].z.m.logerr[`grafana;"failed to process request: ",e];'e}]; + }; + +annotation:{[rqt] + / annotation endpoint is not yet implemented + .z.m.logwarn[`grafana;"annotation url not yet implemented"]; + :`$"Annotation url nyi"; + }; + +query:{[rqt] + / dispatch a /query request to the timeseries or table builder by target type + rqtype:raze rqt[`targets]`type; + :.h.hy[`json]$[rqtype~"timeserie";tsfunc rqt;tbfunc rqt]; + }; + +search:{[rqt] + / build the grafana dropdown options from the tables available in the process + tabs:tables[]; + symtabs:tabs where sym in'cols each tabs; + timetabs:tabs where timecol in'cols each tabs; + rsp:string tabs; + if[count timetabs; + rsp,:s1:prefix["t";string timetabs]; + rsp,:s2:prefix["g";string timetabs]; + / suffix the numeric columns for the graph and other panel options + rsp,:raze(s2,'del),/:'c1:string {cols[x] where`number=types(0!meta x)`t}each timetabs; + rsp,:raze(prefix["o";string timetabs],'del),/:'c1; + if[count symtabs; + / suffix the distinct syms for the timeseries and other panel options + rsp,:raze(s1,'del),/:'c2:string each finddistinctsyms'[timetabs]; + rsp,:raze(prefix["o";string timetabs],'del),/:'{x[0]cross del,'string finddistinctsyms x 1}each(enlist each c1),'timetabs; + ]; + ]; + :.h.hy[`json].j.j rsp; + }; + +finddistinctsyms:{[x] + / distinct syms seen in table x within the configured lookback window + :?[x;enlist(>;timecol;(-;.z.p;timebackdate));1b;{x!x}enlist sym]sym; + }; + +prefix:{[c;s] + / prefix string c and the delimiter to each string in s + :(c,del),/:s; + }; + +/ ----------------------------------------------------------------------------- +/ fetching the last n ticks +/ ----------------------------------------------------------------------------- + +diskvals:{[x] + / last `ticks` rows of an on-disk partitioned table + c:(count[x]-ticks)+til ticks; + :get'[.Q.ind[x;c]]; + }; + +memvals:{[x] + / last `ticks` rows of an in-memory table + :get'[?[x;enlist(within;`i;count[x]-ticks,0);0b;()]]; + }; + +catchvals:{[x] + / try the on-disk path first, fall back to the in-memory path on error + :@[diskvals;x;{[x;y]memvals x}[x]]; + }; + +/ ----------------------------------------------------------------------------- +/ target parsing helpers +/ ----------------------------------------------------------------------------- + +istype:{[targ;char] + / test whether the target is prefixed with char followed by the delimiter + :(char,del)~2#targ; + }; +isfunc:istype[;"f"]; +istab:istype[;"t"]; + +/ ----------------------------------------------------------------------------- +/ building json responses +/ ----------------------------------------------------------------------------- + +tabresponse:{[colname;coltype;rqt] + / build a table response in the json datasource schema + :.j.j enlist`columns`rows`type!(flip`text`type!(colname;coltype);catchvals rqt;`table); + }; + +tbfunc:{[rqt] + / process a table request and return the json datasource table response + rqt:raze rqt[`targets]`target; + symname:0b; + / strip the type prefix: f.t.func drops 4, f.func drops 2, t.tab leaves the + / table name plus an optional sym + rqt:0!value $[isfunc[rqt]&istab 2_rqt;4_rqt; + isfunc rqt;2_rqt; + istab rqt;[rqt:`$del vs rqt;if[2meta[rqt][timecol;`t];rqt:@[rqt;timecol;+;.z.D]]; + / restrict to the time range requested by grafana + range:"P"$-1_'x[`range]`from`to; + rqt:?[rqt;enlist(within;timecol;range);0b;()]; + / add the milliseconds-since-epoch column grafana expects + rqt:@[rqt;`msec;:;mil rqt timecol]; + / dispatch on the number of arguments and the panel type + $[(2