Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/md/how_to/python/virtual_server.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ Perspective ships with built-in virtual server implementations for:
using the `clickhouse-connect` Python package.
- [**Polars**](./virtual_server/polars.md) — query in-memory Polars DataFrames
using the `polars` Python package.
- [**kdb+**](./virtual_server/kdb.md) — query a q process using the `pykx`
Python package.

You can also [**implement your own**](./virtual_server/custom.md) virtual server
to connect Perspective to any data source by subclassing `VirtualServerHandler`.
207 changes: 207 additions & 0 deletions docs/md/how_to/python/virtual_server/kdb.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
# kdb+ Virtual Server

Perspective provides a built-in virtual server for
[kdb+](https://kx.com/), allowing `<perspective-viewer>` clients to query a q
process over WebSocket.

## Installation

```bash
pip install perspective-python pykx
```

## Usage

Start a q process listening on a port:

```bash
q -p 5001
```

Create a server that exposes its tables to browser clients:

```python
import pykx
import tornado.web
import tornado.ioloop
from perspective.virtual_servers.kdb import KdbVirtualServer
from perspective.handlers.tornado import PerspectiveTornadoHandler

# Connect to q over IPC
conn = pykx.SyncQConnection(host="localhost", port=5001)

# Create virtual server backed by kdb+
server = KdbVirtualServer(conn)

# Serve over WebSocket
app = tornado.web.Application([
(r"/websocket", PerspectiveTornadoHandler, {"perspective_server": server}),
])

app.listen(8080)
tornado.ioloop.IOLoop.current().start()
```

Connect from the browser:

```javascript
const websocket = await perspective.websocket("ws://localhost:8080/websocket");
const table = await websocket.open_table("trades");
document.getElementById("viewer").load(table);
```

Tables in q's root namespace are discoverable, as reported by `tables[]`.

## Requirements of the q process

Views are materialized as globals under a `.psp` namespace, so the connected
handle **must permit global assignment**. A read-only handle (`q -b`) or a
gateway that rejects writes will not work; point the virtual server at a
process you control, which may of course proxy a read-only store.

Views are cleaned up when the UI closes them. If a view leaks — because a
client disconnected uncleanly, say — it remains as a global under `.psp` until
the process restarts.

## Type mapping

q's type system is richer than Perspective's six visual types. Columns whose q
type has no Perspective analogue are cast on the way out, so the schema
Perspective reports always matches the data it receives.

| q type | Perspective | Notes |
| ------------------------------- | ----------- | ---------------------------------- |
| `boolean` | `boolean` | |
| `short`, `int`, `byte` | `integer` | |
| `long`, `real`, `float` | `float` | `long` is 64-bit; `integer` is 32 |
| `symbol`, `char`, string | `string` | |
| `guid` | `string` | cast with `string` |
| `date` | `date` | |
| `month` | `date` | cast with `"d"$` |
| `timestamp` | `datetime` | |
| `datetime` | `datetime` | cast with `"p"$` |
| `time`, `minute`, `second`, `timespan` | `string` | cast with `string` |

### Nulls

q has no validity bitmap — a null long *is* `0Nj`, the minimum 64-bit integer.
The handler translates these sentinels to Arrow nulls so they render as empty
cells rather than as `-9223372036854775808`. The empty symbol `` ` `` is
likewise q's symbol null and arrives in Perspective as null, not as `""`.

Infinities (`0W`, `0w`) are genuine values in q and are passed through.

## Supported features

| Feature | Supported | Notes |
| ------------ | --------- | -------------------------------------------------- |
| Group By | ✔ | `rollup`, `flat` and `total` modes |
| Sort | ✔ | |
| Filter | ✔ | see below |
| Aggregates | ✔ | q's own — see below |
| Expressions | ✔ | written in **q**, not Perspective's expression language |
| Windows | ✔ | q's own — `mdev`, `mcount`, `xprev`, `ema` |
| Split By | ✘ | |

This handler exposes **q's data model**, not a lowest common denominator
shared with the other virtual servers. Aggregate and filter names are q's, and
they mean what q means by them. If you know kdb+, the menus should read as
kdb+; if you are moving a saved layout from the DuckDB virtual server, expect
to re-pick aggregates.

### Aggregates

| q aggregate | Applies to | Notes |
| ----------- | ---------- | ----- |
| `sum` `avg` `min` `max` `count` `first` `last` | numeric | `sum` over a boolean column counts the trues |
| `count distinct` | any | q's `count distinct` |
| `med` | numeric | median |
| `dev` / `sdev` | numeric | **population** / **sample** standard deviation |
| `var` / `svar` | numeric | **population** / **sample** variance |
| `prd` | numeric | product |
| `any` / `all` | numeric, boolean | |
| `wavg` / `wsum` | numeric | **weighted** by a second column — `Quantity wavg Sales` |
| `cor` / `cov` | numeric | correlation / covariance against a second column |

`dev` and `sdev` are offered separately because in q they are different
statistics, and the same goes for `var` and `svar`. Nothing here is renamed to
match another backend: there is no `stddev`, no `median`, no `product`.

`wavg`, `wsum`, `cor` and `cov` take a second column, and appear in the
aggregate menu as a submenu of the columns they can pair with.

### Filters

Filter *operators* — `==`, `!=`, `<`, `>`, `<=`, `>=` — are Perspective's
spelling of q's `=`, `<>`, `<`, `>`, `<=`, `>=` and mean the same thing. The
*predicates* are q's:

| Filter op | q |
| --------- | - |
| `like` | q's `like`, taking **q's** pattern language — `*` and `?`. `%` and `_` are literal characters, not wildcards |
| `in` / `not in` | q's `in` over a vector of values |

Ordering comparisons (`<`, `>`, `<=`, `>=`) on string columns compare
lexicographically as symbols, which is q's ordering for `symbol` columns and
the intuitive one for the char-list types.

> If you are used to the DuckDB virtual server, note that `like` patterns are
> **not** translated: `"Bos%"` matches a literal percent sign here. Write
> `"Bos*"`.

`within` is absent despite being idiomatic q, because Perspective's filter UI
infers an operator's operand count from a fixed list of names and would render
a two-operand range filter as a single value box.

### Expressions

Expressions are q, passed through to the q process verbatim — there is no
translation from Perspective's ExprTK-style expression language, so what you
write in the expression editor is q:

```q
Sales * 0.9
```

```q
10 xbar Sales
```

```q
upper City
```

An expression may reference any column of the source table by name, provided
that name is a q identifier. Expressions are validated by q as you type: the
error q reports is the error the editor shows.

An expression's type is resolved by q — you do not declare it — and it behaves
like any other column thereafter, so it can be grouped by, sorted, filtered
and aggregated. An expression whose alias matches a source column shadows it,
matching the SQL virtual servers.

> **Expressions execute in your q process.** They are passed through
> unmodified, so an expression can call anything q can — including your own
> functions and `system`, which reaches the shell. This is the same trust model
> as the DuckDB virtual server, which inlines SQL fragments, but q's reach is
> wider. Only expose a kdb+ virtual server to clients you would grant query
> access to that process, and disable `expressions` in `get_features` if that
> is not true of your deployment.

### Window functions

Window columns are q's own moving and running primitives, under their q names.
A q developer already knows what `mdev` computes; it is not renamed to
`stddev`, which would both rename it and misdescribe it (`mdev` is a
*population* deviation, SQL's `STDDEV_SAMP` a sample one).

| Window aggregate | q |
| ---------------- | - |
| `msum` `mavg` `mmin` `mmax` | the moving verbs; a cumulative frame takes the running form (`sums`, `avgs`, `mins`, `maxs`) |
| `mcount` | moving count of non-nulls |
| `mdev` | moving **population** deviation |
| `mvar` | its square — q has no moving-variance primitive |
| `first` | the earliest row in the frame |
| `xprev` / `xnext` | shift back / forward by an offset |
| `deltas` | difference from the previous row |
| `ema` | exponential moving average — **rejected by the SQL virtual servers**, which have no recursive `OVER` equivalent |
36 changes: 36 additions & 0 deletions examples/python-kdb-virtual/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
# ┃ ██████ ██████ ██████ █ █ █ █ █ █▄ ▀███ █ ┃
# ┃ ▄▄▄▄▄█ █▄▄▄▄▄ ▄▄▄▄▄█ ▀▀▀▀▀█▀▀▀▀▀ █ ▀▀▀▀▀█ ████████▌▐███ ███▄ ▀█ █ ▀▀▀▀▀ ┃
# ┃ █▀▀▀▀▀ █▀▀▀▀▀ █▀██▀▀ ▄▄▄▄▄ █ ▄▄▄▄▄█ ▄▄▄▄▄█ ████████▌▐███ █████▄ █ ▄▄▄▄▄ ┃
# ┃ █ ██████ █ ▀█▄ █ ██████ █ ███▌▐███ ███████▄ █ ┃
# ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
# ┃ Copyright (c) 2017, the Perspective Authors. ┃
# ┃ ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┃
# ┃ This file is part of the Perspective library, distributed under the terms ┃
# ┃ of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). ┃
# ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

FROM --platform=linux/386 i386/debian:bookworm-slim

RUN apt-get update \
&& apt-get install -y --no-install-recommends unzip \
&& rm -rf /var/lib/apt/lists/*

# kdb+ is not redistributable, so it is not downloaded here. Fetch the 32-bit
# Linux archive from https://kx.com yourself and leave it beside this file:
#
# examples/python-kdb-virtual/l32.zip
#
# Note the 32-bit terms permit development and proof-of-concept use, but not
# commercial production use.
COPY l32.zip /tmp/l32.zip
RUN unzip -q /tmp/l32.zip -d /opt && rm /tmp/l32.zip

ENV QHOME=/opt/q

EXPOSE 5001

# `-p` listens on all interfaces, so the port maps out to the host. The
# example's server loads the superstore table over IPC on startup, so this
# process starts empty.
CMD ["/opt/q/l32/q", "-p", "5001"]
31 changes: 31 additions & 0 deletions examples/python-kdb-virtual/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# kdb+ Virtual Server example

Serves a `<perspective-viewer>` backed by a **kdb+** process, via the
[kdb+ virtual server](../../docs/md/how_to/python/virtual_server/kdb.md).

## Prerequisites

A running q process listening on a port. Nothing needs to be in it — this
example loads the superstore dataset over IPC on startup.

```bash
pip install pykx tornado
```

[PyKX](https://code.kx.com/pykx/) connects over IPC in its **unlicensed
mode**, so no kdb+ license is required by this process — only by the q you
connect to.

## Running

```bash
pnpm start
```

Then open <http://localhost:3000>.

Point elsewhere with `PSP_KDB_HOST` / `PSP_KDB_PORT`:

```bash
PSP_KDB_PORT=5010 pnpm start
```
29 changes: 29 additions & 0 deletions examples/python-kdb-virtual/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no" />
<link rel="stylesheet" crossorigin="anonymous" href="/node_modules/@perspective-dev/viewer/dist/css/themes.css" />
<style>
perspective-viewer {
position: absolute;
inset: 0;
}
</style>
</head>
<body>
<perspective-viewer id="viewer" ,> </perspective-viewer>
<script type="module">
import "/node_modules/@perspective-dev/viewer/dist/cdn/perspective-viewer.js";
import "/node_modules/@perspective-dev/viewer-datagrid/dist/cdn/perspective-viewer-datagrid.js";
import "/node_modules/@perspective-dev/viewer-charts/dist/cdn/perspective-viewer-charts.js";
import perspective from "/node_modules/@perspective-dev/client/dist/cdn/perspective.js";
const viewer = document.getElementById("viewer");

// Create a client that expects a Perspective server to accept
// Websocket connections at the specified URL.
const websocket = await perspective.websocket("ws://localhost:3000/websocket");
viewer.load(websocket);
viewer.restore({table: "data_source_one"})
</script>
</body>
</html>
21 changes: 21 additions & 0 deletions examples/python-kdb-virtual/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "python-kdb-virtual",
"private": true,
"version": "5.1.0",
"description": "An example of streaming a `perspective-python` server to the browser.",
"scripts": {
"start": "PYTHONPATH=../../python/perspective python3 server.py"
},
"keywords": [],
"license": "Apache-2.0",
"dependencies": {
"@perspective-dev/client": "workspace:^",
"@perspective-dev/viewer": "workspace:^",
"@perspective-dev/viewer-charts": "workspace:^",
"@perspective-dev/viewer-datagrid": "workspace:^",
"superstore-arrow": "catalog:"
},
"devDependencies": {
"npm-run-all": "catalog:"
}
}
Loading
Loading