diff --git a/docs/js-udf.md b/docs/js-udf.md index 52861b08..1944d98a 100644 --- a/docs/js-udf.md +++ b/docs/js-udf.md @@ -5,6 +5,8 @@ Timeplus supports JavaScript-based UDF running in the SQL engine. You can develo ## Register a JS UDF via SQL {#ddl} Please check [CREATE FUNCTION](/sql-create-function#javascript-udf) page for the SQL syntax. +Since Timeplus Enterprise 3.2.1, a server can refuse new JavaScript UDFs entirely by setting [`enable_javascript_udf: false`](/server_config#enable_javascript_udf) in `timeplusd.yaml`. Creation then fails with *JavaScript UDF creation is disabled by server config `enable_javascript_udf`*; UDFs already registered keep working. + ## Register a JS UDF via Web Console {#register} 1. Open "UDFs" from the navigation menu on the left, and click the 'New UDF' button. diff --git a/docs/named-collection.md b/docs/named-collection.md index e942f0b5..d7c1d235 100644 --- a/docs/named-collection.md +++ b/docs/named-collection.md @@ -77,6 +77,7 @@ Named collections can be used in: 2. External table DDL settings 3. Input DDL settings 4. Disk creation arguments. +5. [Python UDF](/py-udf) initialization parameters, since Timeplus Enterprise 3.3.1. For external stream / external table / input, named collection can specify any setting value other than `type`. @@ -165,3 +166,31 @@ Create a S3 disk. ```sql CREATE DISK s3_disk1 disk(named_collection=s3_config); ``` + +### Python UDF Initialization + +Keep a credential used by a [Python UDF](/py-udf) out of the UDF definition. The collection supplies the `init_function_parameters` key, which is passed to the UDF's init hook when the module loads: + +```sql +CREATE NAMED COLLECTION nc_udf_init AS + init_function_parameters = '{"api_key":"s3cr3t"}' NOT OVERRIDABLE; +``` + +Reference it from the UDF: + +```sql +CREATE OR REPLACE FUNCTION call_api(x string) RETURNS string LANGUAGE PYTHON AS $$ +import json +API_KEY = '' + +def _tp_init(params): + global API_KEY + API_KEY = json.loads(params)['api_key'] + +def call_api(xs): + return [API_KEY + ':' + x.decode('utf-8') for x in xs] +$$ SETTINGS init_function_name = '_tp_init', + named_collection = 'nc_udf_init'; +``` + +`SHOW CREATE FUNCTION call_api` reveals only the collection name, never the value. Creating such a UDF requires the `NAMED COLLECTION` privilege. See [Initialization hook](/py-udf#init_hook) for details. diff --git a/docs/py-udf.md b/docs/py-udf.md index b20e87e0..c46d4ae4 100644 --- a/docs/py-udf.md +++ b/docs/py-udf.md @@ -1,6 +1,10 @@ # Python UDF -In addition to [Remote UDF](/remote-udf) and [JavaScript UDF](/js-udf), starting from [v2.7](/enterprise-v2.7), Timeplus Enterprise also supports Python-based UDF, as a feature in technical preview. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with the embedded Python 3.10 runtime in Timeplus core engine. No need to deploy extra server/service for the UDF. +In addition to [Remote UDF](/remote-udf) and [JavaScript UDF](/js-udf), starting from [v2.7](/enterprise-v2.7), Timeplus Enterprise also supports Python-based UDF, as a feature in technical preview. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with the embedded Python runtime in Timeplus core engine. No need to deploy extra server/service for the UDF. + +:::info Python 3.14 since Timeplus Enterprise 3.3.1 +Timeplus Enterprise 3.3.1 upgrades the embedded interpreter from **Python 3.10 to Python 3.14 (free-threaded, a.k.a. `cp314t`)**, and stops bundling any third-party Python packages. If you are upgrading from 3.2.x or earlier, read [Upgrading to the Python 3.14 runtime](#upgrade_314) before you upgrade — packages you relied on must be reinstalled, and code that mutates module-level state now needs explicit locking. +::: For visual learners, please watch the following video: @@ -176,6 +180,106 @@ class getMax: $$; ``` +## Initialization hook {#init_hook} + +Available since Timeplus Enterprise 3.3.1. + +Some UDFs need one-time setup before the first row is processed — loading a model, compiling a regex, opening a connection, or reading a credential. A Python UDF can declare an **init hook**: a function in the same code block that Timeplus calls once, when the module is loaded, before any UDF invocation (and, for a UDAF, before the class is instantiated). + +Declare it with `SETTINGS init_function_name`: + +```sql +CREATE OR REPLACE FUNCTION tag_value(x string) RETURNS string LANGUAGE PYTHON AS $$ +TAG = '' + +def _tp_init(): + global TAG + TAG = 'ready' + +def tag_value(xs): + return [TAG + ':' + x.decode('utf-8') for x in xs] +$$ SETTINGS init_function_name = '_tp_init'; +``` + +The hook name is validated at `CREATE` time: if the function is not defined in the UDF source, the statement is rejected with `is not defined in the UDF source`. These settings are Python-only — using them on a JavaScript UDF fails with `only supported for Python UDFs`. + +### Passing parameters to the hook {#init_params} + +`init_function_parameters` passes a **single string** as the hook's only argument. Encode structured configuration as JSON and parse it in the hook: + +```sql +CREATE OR REPLACE FUNCTION greet(x string) RETURNS string LANGUAGE PYTHON AS $$ +import json +PREFIX = '' + +def _tp_init(params): + global PREFIX + PREFIX = json.loads(params).get('prefix', '') + +def greet(xs): + return [PREFIX + ':' + x.decode('utf-8') for x in xs] +$$ SETTINGS init_function_name = '_tp_init', + init_function_parameters = '{"prefix":"hello"}'; +``` + +When no parameter source is configured, the hook is called **with no arguments** — so write `def _tp_init():` in that case and `def _tp_init(params):` when you pass parameters. + +### Reading parameters from a named collection {#init_named_collection} + +`init_function_parameters` is stored verbatim in the UDF definition and is echoed back by `SHOW CREATE FUNCTION`, which makes it a poor place for a secret. Use a [named collection](/named-collection) instead: point the UDF at a collection and Timeplus reads the collection's `init_function_parameters` key at module-load time. + +```sql +CREATE NAMED COLLECTION nc_udf_init AS + init_function_parameters = '{"api_key":"s3cr3t"}' NOT OVERRIDABLE; + +CREATE OR REPLACE FUNCTION call_api(x string) RETURNS string LANGUAGE PYTHON AS $$ +import json +API_KEY = '' + +def _tp_init(params): + global API_KEY + API_KEY = json.loads(params)['api_key'] + +def call_api(xs): + return [API_KEY + ':' + x.decode('utf-8') for x in xs] +$$ SETTINGS init_function_name = '_tp_init', + named_collection = 'nc_udf_init'; +``` + +`SHOW CREATE FUNCTION call_api` shows the `init_function_name` and `named_collection` settings but **not** the secret — only the collection name is stored in the UDF. + +The same settings work on `CREATE AGGREGATE FUNCTION`, where the hook runs before the aggregation class is constructed, so `__init__` can rely on whatever the hook set up. + +Rules and behavior: + +* `init_function_parameters` and `named_collection` both require `init_function_name`; without it the statement is rejected. +* They are **mutually exclusive** — configure only one parameter source. +* Creating the UDF requires the `NAMED COLLECTION` privilege on the referenced collection. An ungranted user gets `ACCESS_DENIED`, and gets it whether or not the collection exists, so the error does not leak which collections are defined. +* The collection must exist at `CREATE` time. +* If the collection has no `init_function_parameters` key, that is not an error — the hook is simply called with no arguments. +* Values are resolved when the module loads, **not** on every call. Editing or rotating a named collection does not propagate to already-running queries or materialized views; drop and recreate the materialized view to pick up a new value. +* If the collection is dropped after the UDF is created, the next module load fails cleanly with `Named collection '…' required by UDF '…' does not exist`. +* If the hook itself raises, the query fails with the Python exception and the partially initialized module is discarded, so a later call re-runs the hook from scratch. + +## Turning the runtime off {#enable_flag} + +Available since **Timeplus Enterprise 3.2.1**. Environments that do not want operators shipping arbitrary Python into the engine can refuse new Python UDFs with a server config flag in `timeplusd.yaml`: + +```yaml +enable_python_udf: false # default: true +enable_javascript_udf: false # default: true, gates JavaScript UDF the same way +``` + +The flag is picked up by `SYSTEM RELOAD CONFIG` — no restart needed. While it is `false`, both `CREATE FUNCTION ... LANGUAGE PYTHON` and `CREATE AGGREGATE FUNCTION ... LANGUAGE PYTHON` fail with: + +``` +Python UDF creation is disabled by server config `enable_python_udf` +``` + +The same check runs on the REST UDF endpoint, so registering the function from the web console fails in the same way. + +This gates **creation only**. Python UDFs that already exist keep running, and package management commands are unaffected. To take an existing function out of service, drop it. Setting the flag on all nodes is what makes it meaningful — it is read from each node's own config, not replicated cluster-wide. + ## Manage Python Libraries {#python_libs} Starting from Proton/Timeplus Enterprise 3.0, manage Python UDF packages directly via SQL `SYSTEM` commands. This is the only supported flow on 3.0+. The 2.x methods below are not supported on 3.0. @@ -193,6 +297,15 @@ SYSTEM INSTALL PYTHON PACKAGE 'requests==2.32.3'; -- Alternative form with separate version literal SYSTEM INSTALL PYTHON PACKAGE 'requests' '2.32.3'; +-- Install several packages in one task from inline requirements text (3.1.2+) +SYSTEM INSTALL PYTHON PACKAGE REQUIREMENTS 'requests==2.32.3 +pydantic>=2.0'; + +-- Install from a private mirror, falling back to PyPI (3.1.2+) +SYSTEM INSTALL PYTHON PACKAGE 'internal-lib==1.4.0' + INDEX_URL 'https://mirror.example.com/simple' + EXTRA_INDEX_URL 'https://pypi.org/simple'; + -- List installed packages SYSTEM LIST PYTHON PACKAGES; @@ -201,8 +314,9 @@ SYSTEM UNINSTALL PYTHON PACKAGE 'requests'; ``` Notes: -- Applies to Proton/Enterprise 3.0+ with Python UDF enabled (Python 3.10). +- Applies to Proton/Enterprise 3.0+ with Python UDF enabled (Python 3.14 since 3.3.1, Python 3.10 before that). - Cluster-wide operation; requires `SYSTEM RELOAD CONFIG` privilege. +- `REQUIREMENTS`, `INDEX_URL` and `EXTRA_INDEX_URL` require 3.1.2+ on every node. Requirements text takes one package per line — pip options such as `-r` or `--index-url` inside the text are rejected, pass them as clauses instead. See [SYSTEM PYTHON PACKAGES](/sql-system-python-packages#requirements) for the full rules. - `SYSTEM LIST PYTHON PACKAGES` returns columns `package_name`, `version`. - Install/uninstall runs asynchronously. Check status via `system.python_package_tasks`: ```sql @@ -221,55 +335,48 @@ Permissions: See more: /sql-system-python-packages +### Declarative package management with `python_requirements` {#python_requirements} + +Available since Timeplus Enterprise 3.3.1. `SYSTEM INSTALL PYTHON PACKAGE` installs into the local user site-packages of each node, which does not survive a reschedule on a node without a persistent volume. For clusters — and especially for ephemeral compute nodes — declare your packages in a `requirements.txt` on S3 instead and let every node reconcile against it. + +Add a `python_requirements` section to `timeplusd.yaml`: + +```yaml +python_requirements: + url: https://my-bucket.s3.us-west-2.amazonaws.com/proton/requirements.txt + # How often (seconds) to re-check the file for changes after a successful reconcile. + # 0 disables polling (reconcile on startup only). Default: 300. + poll_interval_sec: 300 + # Credentials are optional; without them the environment chain + # (IRSA / instance profile, AWS_* environment variables) is used. + # access_key_id: ACCESS_KEY_ID + # secret_access_key: SECRET_ACCESS_KEY + # region: us-west-2 + # Optional pip index overrides, e.g. an internal mirror. + # index_url: https://pypi.org/simple + # extra_index_url: https://my-mirror.example.com/simple +``` + +How it behaves: +* Every node fetches the file on startup, and again every `poll_interval_sec`, so edits to the file roll out without a restart. +* The file is the durable source of truth: it restores packages on ephemeral compute nodes after a reschedule, and keeps data nodes with persistent volumes in sync. +* **Reconcile only installs.** Packages removed from the file are *not* uninstalled — do that manually with `SYSTEM UNINSTALL PYTHON PACKAGE`. +* `SYSTEM INSTALL`/`UNINSTALL PYTHON PACKAGE` still works as a manual escape hatch, but packages installed that way are not recorded in the file and do not survive a reschedule on nodes without persistent volumes. +* Pin exact versions (`package==x.y.z`) so all nodes converge on identical environments. + ### Built-in Libraries -By default, Timeplus ships a clean Python 3.10 environment, plus the following essential libraries: - -```json -[ - { "name": "annotated-types", "version": "0.7.0" }, - { "name": "anyio", "version": "4.9.0" }, - { "name": "asyncer", "version": "0.0.8" }, - { "name": "autogen", "version": "0.7.3" }, - { "name": "certifi", "version": "2025.1.31" }, - { "name": "charset-normalizer", "version": "3.4.1" }, - { "name": "diskcache", "version": "5.6.3" }, - { "name": "distro", "version": "1.9.0" }, - { "name": "docker", "version": "7.1.0" }, - { "name": "exceptiongroup", "version": "1.2.2" }, - { "name": "fast-depends", "version": "2.4.12" }, - { "name": "h11", "version": "0.14.0" }, - { "name": "httpcore", "version": "1.0.7" }, - { "name": "httpx", "version": "0.28.1" }, - { "name": "idna", "version": "3.10" }, - { "name": "jiter", "version": "0.9.0" }, - { "name": "numpy", "version": "2.2.4" }, - { "name": "openai", "version": "1.68.2" }, - { "name": "packaging", "version": "24.2" }, - { "name": "pip", "version": "22.0.2" }, - { "name": "proton-driver", "version": "0.2.13" }, - { "name": "pyautogen", "version": "0.7.3" }, - { "name": "pydantic", "version": "2.10.6" }, - { "name": "pydantic_core", "version": "2.27.2" }, - { "name": "python-dotenv", "version": "1.0.1" }, - { "name": "pytz", "version": "2025.1" }, - { "name": "regex", "version": "2024.11.6" }, - { "name": "requests", "version": "2.32.3" }, - { "name": "setuptools", "version": "78.0.2" }, - { "name": "setuptools-scm", "version": "8.2.0" }, - { "name": "six", "version": "1.17.0" }, - { "name": "sniffio", "version": "1.3.1" }, - { "name": "sseclient-py", "version": "1.8.0" }, - { "name": "termcolor", "version": "2.5.0" }, - { "name": "tiktoken", "version": "0.9.0" }, - { "name": "timeplus-neutrino", "version": "0.0.6" }, - { "name": "tomli", "version": "2.2.1" }, - { "name": "tqdm", "version": "4.67.1" }, - { "name": "typing_extensions", "version": "4.12.2" }, - { "name": "tzlocal", "version": "5.3.1" }, - { "name": "urllib3", "version": "2.3.0" }, - { "name": "websockets", "version": "14.2" }, - { "name": "wheel", "version": "0.37.1" } -] +Timeplus ships a **clean Python 3.14 environment**: the Python standard library, plus `pip` and `truststore` so that package installation works out of the box. **No third-party libraries are bundled.** + +:::warning Changed in 3.3.1 +Timeplus Enterprise 3.2.x and earlier bundled ~40 packages, including `numpy`, `requests`, `openai`, `pydantic`, `proton-driver` and `timeplus-neutrino`. **None of them ship in 3.3.1 or later.** Any UDF that imports one of them fails with `ModuleNotFoundError` until you install it explicitly — see [Upgrading to the Python 3.14 runtime](#upgrade_314). + +Bundling was dropped so that the runtime no longer pins you to versions Timeplus happened to choose, and so that the shipped artifact carries no third-party CVE surface you did not ask for. Install exactly what your UDFs need with [`SYSTEM INSTALL PYTHON PACKAGE`](#install_sql) or [`python_requirements`](#python_requirements). +::: + +To see what is currently installed on a node: + +```sql +SYSTEM LIST PYTHON PACKAGES; ``` ### Verified Libraries {#verified_libs} @@ -343,8 +450,64 @@ curl -H "x-timeplus-user: theUser" -H "x-timeplus-key:thePwd" -X DELETE http://l ### Update Python Libraries {#update_lib} There is no in-place update. Uninstall then install the desired version. +## Concurrency and the free-threaded runtime {#free_threading} + +Since Timeplus Enterprise 3.3.1 the embedded interpreter is a **free-threaded** build of Python 3.14 ([PEP 703](https://peps.python.org/pep-0703/)): it is compiled with `Py_GIL_DISABLED`, so the Global Interpreter Lock is gone and UDFs from concurrent queries genuinely run in parallel on multiple threads. This is what removes the single-interpreter bottleneck that Python UDFs had on 3.10 — but it also means **the GIL no longer accidentally protects your Python state**. + +At startup the server logs which build is in use, for example: + +``` +Embedded Python 3.14.6 interpreter is initializing: embedded_free_threaded=true, ... +``` + +What is safe and what is not: + +* **Module-level state inside the UDF body is per query.** Each query gets its own module object for the `$$ ... $$` code, so a global defined there is not shared with other queries. This is safe. +* **Module-level state inside an *imported* helper module is shared.** Imports go through `sys.modules`, which is interpreter-global, so every concurrent query on every thread sees the *same* module object and the same globals. Read-modify-write on such a global is a data race, and updates are silently lost. + +The failure is silent — no exception, just wrong numbers. In a measured run of 8 concurrent queries each doing 5000 increments of a counter in an imported module, an unprotected counter finished at roughly 18,000 instead of 40,000; the same code with a `threading.Lock` finished at exactly 40,000. + +So if a helper module holds mutable state, guard it: + +```python +# my_helpers.py, installed as a package or placed on the interpreter path +import threading + +_lock = threading.Lock() +_counter = 0 + +def bump(): + global _counter + with _lock: # required on the free-threaded runtime + _counter += 1 + return _counter +``` + +Read-only module state (lookup tables, compiled regexes, loaded models) needs no lock. The place to audit is any shared, *mutated* global — counters, caches, accumulators, and connection pools that are not themselves thread-safe. + +## Upgrading to the Python 3.14 runtime {#upgrade_314} + +When upgrading from Timeplus Enterprise 3.2.x or earlier to 3.3.1+: + +1. **Replace the whole runtime, not just `timeplusd`.** The embedded interpreter loads `libpython3.14t` from the Python bundle that ships with the release, and the server refuses to start if the runtime is a GIL-enabled build. Follow the standard [bare metal upgrade](/bare-metal-install) procedure — stop the service, replace both `bin/` and `lib/`, then start it again. The data folder needs no migration; UDF definitions, streams and checkpoints all survive. +2. **Reinstall the packages your UDFs import.** Nothing third-party is bundled anymore. Inventory your UDF `import` statements first, then install them, ideally by declaring them in [`python_requirements`](#python_requirements) so all nodes converge: + ```sql + SYSTEM INSTALL PYTHON PACKAGE 'proton-driver==0.3.0'; + SYSTEM INSTALL PYTHON PACKAGE 'numpy'; + ``` + No server restart is needed after installing. +3. **Check that the packages have free-threaded wheels.** The interpreter is `cp314t`, so a package needs a `cp314t` (or pure-Python) wheel; otherwise pip has to build it from source, which requires a toolchain on the node. `proton-driver` 0.3.0 publishes a `cp314t` wheel and works on the free-threaded runtime. +4. **Audit shared mutable state in imported helper modules** and add locking — see [Concurrency and the free-threaded runtime](#free_threading). +5. **Rolling back** is a data-folder copy: with the service stopped, copy the data folder aside before you upgrade. Restoring that copy and reinstalling the old release brings 3.2.x back up intact. + +:::tip +Right after `SYSTEM INSTALL PYTHON PACKAGE`, the very first UDF call can still fail with `ModuleNotFoundError` because the interpreter cached the (previously missing) site-packages directory at startup. Simply retry the query — no restart is needed. +::: + ## Limitations - Linux deployments require Glibc 2.35+. -- Python 3.10 only for the embedded runtime. +- The embedded runtime is Python 3.14 free-threaded (`cp314t`) since Timeplus Enterprise 3.3.1, and Python 3.10 in earlier versions. The version is not configurable, and packages must be compatible with it. +- No third-party packages are bundled since 3.3.1; install what you need via SQL or `python_requirements`. +- Shared state in imported modules is not protected by a GIL — see [Concurrency and the free-threaded runtime](#free_threading). - Some libraries may require OS/system dependencies. - On 3.0+, use SQL `SYSTEM` commands; on 2.x, use REST or `timeplusd python -m pip`. diff --git a/docs/server_config.md b/docs/server_config.md index 74365dcc..ba24d0db 100644 --- a/docs/server_config.md +++ b/docs/server_config.md @@ -276,6 +276,20 @@ LEFT JOIN tableB ON tableA.id=tableB.id SETTINGS join_max_buffered_bytes=8589934592; ``` +#### enable_python_udf +Type: bool + +Default: true + +Available since Timeplus Enterprise 3.2.1. When set to false, this node rejects `CREATE FUNCTION`/`CREATE AGGREGATE FUNCTION` with `LANGUAGE PYTHON` — via SQL or the REST endpoint — with the error *Python UDF creation is disabled by server config `enable_python_udf`*. Existing Python UDFs continue to run; the flag gates creation only. It is applied by `SYSTEM RELOAD CONFIG` and is per-node, so set it on every node. See [Python UDF](/py-udf#enable_flag). + +#### enable_javascript_udf +Type: bool + +Default: true + +Available since Timeplus Enterprise 3.2.1. The JavaScript counterpart of `enable_python_udf`, rejecting creation of `LANGUAGE JAVASCRIPT` UDFs and UDAFs with the error *JavaScript UDF creation is disabled by server config `enable_javascript_udf`*. + #### logger ```yaml logger: diff --git a/docs/shared/python-external-stream-write.md b/docs/shared/python-external-stream-write.md index 8030d887..b55bfb12 100644 --- a/docs/shared/python-external-stream-write.md +++ b/docs/shared/python-external-stream-write.md @@ -45,6 +45,62 @@ CREATE MATERIALIZED VIEW high_value_alerts INTO py_alert_sink AS The materialized view feeds chunks into the sink as they are produced; each chunk becomes one call to `py_alert_sink`. +### Flushing buffered writes {#flush} + +Available since **Timeplus Enterprise 3.3.1**. + +Sinks that batch — accumulating rows in Python and shipping them in one API call, one file, or one bulk insert — need a moment to push what is still in the buffer. `flush_function_name` names a zero-argument function that Timeplus calls for you: + +* **On every checkpoint**, so a long-running materialized view sink does not hold rows indefinitely. +* **Once when the query closes**, immediately *before* the deinit function, so a graceful shutdown does not drop buffered rows. + +```sql +CREATE EXTERNAL STREAM py_batched_sink (event_id string, body string) +AS $$ +import builtins, json, urllib.request + +def open_client(config): + builtins._tp_batch = {"url": json.loads(config)["url"], "rows": []} + +def collect(event_id, body): + builtins._tp_batch["rows"].extend( + {"id": eid.decode(), "body": b.decode()} for eid, b in zip(event_id, body) + ) + +def flush_batch(): + batch = builtins._tp_batch + if not batch["rows"]: + return + req = urllib.request.Request( + batch["url"], + data=json.dumps(batch["rows"]).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + urllib.request.urlopen(req).read() + batch["rows"] = [] + +def close_client(): + if hasattr(builtins, "_tp_batch"): + del builtins._tp_batch +$$ +SETTINGS + type = 'python', + init_function_name = 'open_client', + init_function_parameters = '{"url":"https://hooks.example.com/bulk"}', + write_function_name = 'collect', + flush_function_name = 'flush_batch', + deinit_function_name = 'close_client'; +``` + +Notes: + +* The flush function takes **no arguments** and its return value is ignored. Make it safe to call when the buffer is empty — Timeplus does not know whether you have anything pending. +* It runs at most once during close, even if close is reached through both an explicit finish and session teardown, so `flush` followed by `deinit` is the exact order to rely on. +* It is a **sink-only** hook. A stream that is only read from never calls it, no matter what `flush_function_name` is set to. +* The name must exist in the Python body. A missing function fails the INSERT with `UDF_INTERNAL_ERROR`, and unlike `init_function_parameters` this is *not* caught at `CREATE` time. +* If flush raises during close, the exception surfaces from the query and deinit still runs. + ### Custom protocol example: webhook POST Load the destination URL in init, reuse that configuration for every chunk, and clear it in deinit. Init parameters carry the URL so the Python body is reusable across environments. To pool an actual HTTP connection, swap `urllib` for a session-aware client (for example `requests.Session()`) and stash the session itself on `builtins`. diff --git a/docs/shared/python-external-stream.md b/docs/shared/python-external-stream.md index e67a3a33..49e369c5 100644 --- a/docs/shared/python-external-stream.md +++ b/docs/shared/python-external-stream.md @@ -20,6 +20,9 @@ def init_fn(config): # optional def deinit_fn(): # optional ... + +def flush_fn(): # optional, sink only + ... $$ SETTINGS type = 'python', -- required @@ -28,6 +31,7 @@ SETTINGS init_function_name = '..', init_function_parameters = '..', -- requires init_function_name deinit_function_name = '..', + flush_function_name = '..', -- sink only, 3.3.1+ mode = 'auto' -- 'auto' (default), 'streaming', or 'batch' ``` @@ -39,6 +43,7 @@ SETTINGS * **init_function_name**: name of a Python function called once before read/write processing begins. Use it to open connections, warm caches, or prepare state for the entry function to consume. * **init_function_parameters**: a single string passed as the only argument to the init function. Any format works (JSON, `key=value`, or a plain string) — parsing is up to your Python code. Requires `init_function_name`; otherwise the stream fails to create with `Setting 'init_function_parameters' requires 'init_function_name' to be configured`. * **deinit_function_name**: name of a Python function called once after read/write processing completes, for cleanup. +* **flush_function_name**: name of a Python function called on every checkpoint and once before deinit, so buffered writes are not lost. Takes no arguments. **Write path only** — it is never called when the stream is read from. Available since **Timeplus Enterprise 3.3.1**. See [Flushing buffered writes](/python-external-stream-sink#flush). * **mode**: Python execution mode — `'auto'` (default), `'streaming'`, or `'batch'`. See [Modes](#modes). ## Modes @@ -59,7 +64,8 @@ Each query that reads from or writes to a Python External Stream creates its own 2. Local API credential globals are injected into the module (see [Local API credentials](#local-api-credentials)). 3. If `init_function_name` is set, the init function is called once. When `init_function_parameters` is non-empty, it is passed as the only argument; otherwise init receives no arguments. 4. The read or write entry function is called as data flows. -5. When the query ends — normally or via cancellation — `deinit_function_name`, if set, is called. +5. On the write path only, if `flush_function_name` is set it is called at every checkpoint, and once more when the query closes — before deinit. +6. When the query ends — normally or via cancellation — `deinit_function_name`, if set, is called. Each query gets its own module, so ordinary module globals created by the DDL body are not reused across queries. If you stash state on Python's `builtins` module, use a stream-specific attribute name and remove it in deinit; `builtins` is shared by the embedded interpreter, so leftover attributes can be visible to later Python sessions in the same server process. Treat clients or caches opened in init as per-query resources and close them in deinit. diff --git a/docs/sql-create-external-stream.md b/docs/sql-create-external-stream.md index 6bc5948d..050302d2 100644 --- a/docs/sql-create-external-stream.md +++ b/docs/sql-create-external-stream.md @@ -70,10 +70,11 @@ SETTINGS init_function_name = '..', init_function_parameters = '..', deinit_function_name = '..', + flush_function_name = '..', -- sink only, 3.3.1+ mode = 'auto' -- 'auto' (default), 'streaming', or 'batch' ``` -Available in **Timeplus Enterprise 3.2.2+**. +Available in **Timeplus Enterprise 3.2.2+**. `flush_function_name` requires **3.3.1+**. Please check the [Python External Stream Source](/python-external-stream-source) for read-side settings, generator/batch sources, and lifecycle hooks, and the [Python External Stream Sink](/python-external-stream-sink) for write-side semantics, materialized-view sinks, and custom-protocol examples. diff --git a/docs/sql-create-function.md b/docs/sql-create-function.md index 504a5455..28f68c5a 100644 --- a/docs/sql-create-function.md +++ b/docs/sql-create-function.md @@ -131,7 +131,7 @@ $$; [Learn More](/js-udf) ## Python UDF -starting from v2.7, Timeplus Enterprise also supports Python-based UDF. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with the embedded Python 3.10 runtime in Timeplus core engine. No need to deploy extra server/service for the UDF. +starting from v2.7, Timeplus Enterprise also supports Python-based UDF. You can develop User-defined scalar functions (UDFs) or User-defined aggregate functions (UDAFs) with the embedded Python runtime in Timeplus core engine — Python 3.14 free-threaded since Timeplus Enterprise 3.3.1, Python 3.10 in earlier versions. No need to deploy extra server/service for the UDF. [Learn more](/py-udf) why Python UDF, and how to map the data types in Timeplus and Python, as well as how to manage dependencies. @@ -180,4 +180,31 @@ $$ SETTINGS ... ``` -[Learn More](/py-udf) +### Settings {#py-udf-settings} + +Python UDFs and UDAFs accept the following settings, all available since Timeplus Enterprise 3.3.1. Any other setting name is rejected at creation time, and all three are Python-only — using them with `LANGUAGE JAVASCRIPT` fails. + +| Setting | Description | +| -- | -- | +| `init_function_name` | Name of a function defined in the same code block that Timeplus calls once when the module is loaded, before the first UDF call and before a UDAF class is constructed. Must exist in the source, or the `CREATE` is rejected. | +| `init_function_parameters` | A single string passed as the only argument to the init function. Encode structured configuration as JSON. Requires `init_function_name`. Stored in the UDF definition and visible in `SHOW CREATE FUNCTION` — do not put secrets here. | +| `named_collection` | Name of a [named collection](/named-collection) whose `init_function_parameters` key is passed to the init function instead. Requires `init_function_name`, requires the `NAMED COLLECTION` privilege, and is mutually exclusive with `init_function_parameters`. Only the collection name is stored, so the value stays out of `SHOW CREATE FUNCTION`. | + +If neither parameter source is set, the init function is called with no arguments. + +```sql +CREATE OR REPLACE FUNCTION call_api(x string) RETURNS string LANGUAGE PYTHON AS $$ +import json +API_KEY = '' + +def _tp_init(params): + global API_KEY + API_KEY = json.loads(params)['api_key'] + +def call_api(xs): + return [API_KEY + ':' + x.decode('utf-8') for x in xs] +$$ SETTINGS init_function_name = '_tp_init', + named_collection = 'nc_udf_init'; +``` + +[Learn More](/py-udf) — see [Initialization hook](/py-udf#init_hook) for the full behavior. diff --git a/docs/sql-system-python-packages.md b/docs/sql-system-python-packages.md index 1e366238..e18c1d78 100644 --- a/docs/sql-system-python-packages.md +++ b/docs/sql-system-python-packages.md @@ -14,6 +14,22 @@ SYSTEM INSTALL PYTHON PACKAGE 'requests==2.32.3'; -- Alternative form with separate version literal SYSTEM INSTALL PYTHON PACKAGE 'requests' '2.32.3'; +-- Install several packages at once from inline requirements text (3.1.2+) +SYSTEM INSTALL PYTHON PACKAGE REQUIREMENTS 'requests==2.32.3 +pydantic>=2.0 +# comments and blank lines are ignored +httpx'; + +-- Install from a private mirror instead of PyPI (3.1.2+) +SYSTEM INSTALL PYTHON PACKAGE 'internal-lib==1.4.0' + INDEX_URL 'https://mirror.example.com/simple'; + +-- Fall back to PyPI for anything the private mirror does not carry +SYSTEM INSTALL PYTHON PACKAGE REQUIREMENTS 'internal-lib==1.4.0 +requests==2.32.3' + INDEX_URL 'https://mirror.example.com/simple' + EXTRA_INDEX_URL 'https://pypi.org/simple'; + -- List installed packages (returns: package_name, version) SYSTEM LIST PYTHON PACKAGES; @@ -21,12 +37,48 @@ SYSTEM LIST PYTHON PACKAGES; SYSTEM UNINSTALL PYTHON PACKAGE 'requests'; ``` +:::info Clean environment since 3.3.1 +Timeplus Enterprise 3.3.1 upgraded the embedded interpreter to **Python 3.14 free-threaded** and stopped bundling third-party packages — only the standard library, `pip` and `truststore` ship with the product. Everything your UDFs import must be installed with the commands on this page (or declared via [`python_requirements`](#declarative)). See [Python UDF](/py-udf#upgrade_314) for the upgrade steps. +::: + +## Installing from requirements text {#requirements} + +Available since **Timeplus Enterprise 3.1.2**. + +`SYSTEM INSTALL PYTHON PACKAGE REQUIREMENTS ''` takes the contents of a `requirements.txt` as a single string literal and installs every line in one cluster-wide task. Rules: + +- One package specification per line, using the same PEP 440 syntax as the single-package form. Blank lines and `#` comments are skipped. +- **pip options are rejected.** Any line starting with `-` (`-r nested.txt`, `--index-url ...`, `-e .`) fails with `Requirements line '...' is not supported. Please pass index options via API fields instead` — use the `INDEX_URL` / `EXTRA_INDEX_URL` clauses below. +- The text must contain at least one package specification, and is capped at 1024 effective lines. +- `REQUIREMENTS` cannot be combined with a package-name literal — `SYSTEM INSTALL PYTHON PACKAGE REQUIREMENTS 'requests' '2.0'` is a syntax error. +- The whole batch is tracked as a **single** row in `system.python_package_tasks`, with `package_name` set to the literal `requirements.txt` rather than to any individual package: + ```sql + SELECT status, error_code, error_message + FROM system.python_package_tasks + WHERE package_name = 'requirements.txt' AND operation = 'install' + ORDER BY created_at DESC + LIMIT 1; + ``` + +## Private package indexes {#index-url} + +Available since **Timeplus Enterprise 3.1.2**. Both clauses work with either install form, and may be combined: + +- **INDEX_URL '\'** — replaces the default index, mapping to pip's `--index-url`. At most one. +- **EXTRA_INDEX_URL '\'** — an additional index consulted after the primary one, mapping to pip's `--extra-index-url`. Repeat the clause to pass several. + +Only `http://` and `https://` URLs are accepted; anything else is rejected up front with `Invalid --index-url value '...'. Only http(s) URLs are supported`, before any node runs pip. Credentials embedded in the URL (`https://user:token@mirror.example.com/simple`) are passed through to pip as-is, but the statement is distributed across the cluster and recorded in `system.query_log` like any other query — prefer a mirror that authenticates by network location, or configure the index once in [`python_requirements`](#declarative) rather than repeating the secret in every statement. + +These clauses apply to the statement only; they are not remembered for later installs. + ## Behavior -- Scope: Cluster-wide installation/uninstallation using the UDF runtime’s Python 3.10 environment. +- Scope: Cluster-wide installation/uninstallation using the UDF runtime’s Python environment (Python 3.14 since 3.3.1, Python 3.10 before that). - Permissions: Requires `SYSTEM RELOAD CONFIG` privilege. - Versioning: Accepts PEP 440 specifiers in the first literal (e.g., `>=`, `==`, `~=`). When using the second literal, provide the exact version string. - Install location: Uses pip’s user install under the embedded interpreter; no system-level Python changes. - Async operations: Install/uninstall run asynchronously. Track progress via `system.python_package_tasks`. +- Wheel compatibility: The 3.3.1+ interpreter is `cp314t`, so a package needs a free-threaded (or pure-Python) wheel; otherwise pip builds it from source and needs a toolchain on the node. +- Durability: Packages installed this way live in the node’s local user site-packages. On a node without a persistent volume they are lost on reschedule — use [`python_requirements`](#declarative) for those. Monitor status ```sql @@ -48,6 +100,21 @@ Granting permissions GRANT SYSTEM RELOAD CONFIG ON *.* TO gen; ``` +## Declarative alternative: `python_requirements` {#declarative} + +Since Timeplus Enterprise 3.3.1 you can declare packages in a `requirements.txt` on S3 and have every node reconcile against it, instead of issuing `SYSTEM INSTALL` per node. This is the recommended approach for clusters and for ephemeral compute nodes, where locally installed packages do not survive a reschedule. + +```yaml +# timeplusd.yaml +python_requirements: + url: https://my-bucket.s3.us-west-2.amazonaws.com/proton/requirements.txt + poll_interval_sec: 300 # re-check for changes; 0 = startup only +``` + +Each node fetches the file on startup and then every `poll_interval_sec`, installing anything missing — so edits roll out without a restart. The reconcile only *installs*: removing a line does not uninstall the package, use `SYSTEM UNINSTALL PYTHON PACKAGE` for that. Pin exact versions so all nodes converge. Full options are documented in [Python UDF](/py-udf#python_requirements). + ## Compatibility - Proton/Enterprise 3.0+: Use these SQL commands. This is the only supported method in 3.0+. +- Enterprise 3.1.2+: `REQUIREMENTS`, `INDEX_URL` and `EXTRA_INDEX_URL` clauses. A statement using any of them is dispatched to the cluster with a newer request format, so every node must be on 3.1.2 or later. +- Enterprise 3.3.1+: The embedded runtime is Python 3.14 free-threaded and ships no third-party packages; `python_requirements` is available as a declarative alternative. - Enterprise 2.x: Use REST API or `timeplusd python -m pip` (see /py-udf#install_lib). These legacy methods are not supported on 3.0+.