diff --git a/docs/_static/audit-data-flow.png b/docs/_static/audit-data-flow.png new file mode 100644 index 00000000000..94c3f8a4cae Binary files /dev/null and b/docs/_static/audit-data-flow.png differ diff --git a/docs/_static/audit-filter-hierarchy.png b/docs/_static/audit-filter-hierarchy.png new file mode 100644 index 00000000000..0bff15244b2 Binary files /dev/null and b/docs/_static/audit-filter-hierarchy.png differ diff --git a/docs/_static/audit-log-filter-arch.png b/docs/_static/audit-log-filter-arch.png new file mode 100644 index 00000000000..d21f83f7f84 Binary files /dev/null and b/docs/_static/audit-log-filter-arch.png differ diff --git a/docs/_static/audit-log-filter-modification-lifecycle.png b/docs/_static/audit-log-filter-modification-lifecycle.png new file mode 100644 index 00000000000..358b154ac07 Binary files /dev/null and b/docs/_static/audit-log-filter-modification-lifecycle.png differ diff --git a/docs/_static/audit-logging-strategy.png b/docs/_static/audit-logging-strategy.png new file mode 100644 index 00000000000..ae986f0d5af Binary files /dev/null and b/docs/_static/audit-logging-strategy.png differ diff --git a/docs/audit-log-filter-compression-encryption.md b/docs/audit-log-filter-compression-encryption.md index e472156b1bd..705d2b45841 100644 --- a/docs/audit-log-filter-compression-encryption.md +++ b/docs/audit-log-filter-compression-encryption.md @@ -2,42 +2,42 @@ ## Compression -You can enable compression for any [format](audit-log-filter-formats.md) by setting the `audit_log_filter.compression` system variable when the server starts. +Enable compression for any [format](audit-log-filter-formats.md) with `audit_log_filter.compression` at server startup. -The `audit_log_filter.compression` variable can be either of the following: +Allowed values: -* NONE (no compression) - the default value -* GZIP - uses the GNU Zip compression +* `NONE` — the default. No compression. -If compression and encryption are enabled, the component applies compression before encryption. If you must manually recover a file with both settings, first decrypt the file and then uncompress the file. +* `GZIP` — GNU zip compression. + +With both compression and encryption enabled, the component compresses first and then encrypts. To recover a file manually, decrypt first and then decompress. ## Encryption -You can encrypt any audit log filter file in any [format](audit-log-filter-formats.md). The audit log filter component generates the initial password, but you can use user-defined passwords after that. The component stores the passwords in the keyring, so that feature must be enabled. +Encrypt any audit log format. The component generates the first password. You can rotate to custom passwords afterward. Passwords live in the keyring. Enable a keyring first. -Set the `audit_log_filter.encryption` system variable with the server starts. The allowed values are the following: +Set `audit_log_filter.encryption` at startup. Values: -* NONE - no encryption, the default value -* AES - AES-256-CBC (Cipher Block Chaining) encryption +* `NONE` — the default. No encryption. -The AES uses the 256-bit key size. +* `AES` — AES-256-CBC. -The following audit log filter functions are used with encryption: +AES uses a 256-bit key. -| Function name | Description | -| ----------------- | -------------------- | -| audit_log_encryption_password_set() | Stores the password in the keyring. If encryption is enabled, the function also rotates the log file by renaming the current log file and creating a log file encrypted with the password. | -| audit_log_encryption_password_get() | Invoking this function without an argument returns the current encryption password. An argument that specifies the keyring ID of an archived password or current password returns that password by ID. | +Encryption-related functions: -The `audit_log_filter.password_history_keep_days` variable is used with encryption. If the variable is not zero (0), invoking `audit_log_encryption_password_set()` causes the expiration of archived audit log passwords. +| Function name | Description | +|---|---| +| `audit_log_encryption_password_set()` | Stores a password in the keyring. With encryption on, the function also rotates the log: renames the current file and starts a new file encrypted with the new password. | +| `audit_log_encryption_password_get()` | With no argument, returns the active password. With a keyring ID, returns the archived or current password by ID. | -When the component starts with encryption enabled, the component checks if the keyring has an audit log filter encryption password. If no password is found, the component generates a random password and stores this password in the keyring. Use `audit_log_encryption_password_get()` to review this password. +`audit_log_filter.password_history_keep_days` controls how long archived passwords stay available. When the value is non-zero, calling `audit_log_encryption_password_set()` can expire older keyring entries. -If compression and encryption are enabled, the component applies compression before encryption. If you must manually recover a file with both settings, first decrypt the file and then uncompress the file. +On startup with encryption enabled, the component generates a password and stores the password when none exists. Call `audit_log_encryption_password_get()` to inspect the password. -## Manually uncompressing and decrypting audit log filter files +## Manually uncompress and decrypt audit log filter files -To decrypt an encrypted log file, use the openssl command. For example: +Decrypt with OpenSSL. For example: ```bash openssl enc -d -aes-256-cbc -pass pass:password @@ -46,20 +46,30 @@ openssl enc -d -aes-256-cbc -pass pass:password -out audit.timestamp.log ``` -To execute that command, you must obtain a password and iterations. To do this, use `audit_log_encryption_password_get()`. - -This function gets the encryption password, and the iterations count and returns this data as a JSON-encoded string. For example, if the audit log file name is `audit.20190415T151322.log.20190414T223342-2.enc`, the password ID is `{randomly-generated-alphanumeric-string}` and the keyring ID is `audit-log-20190414T223342-2`. +You need the password and iteration count from `audit_log_encryption_password_get()`. -Get the keyring password: +The function returns JSON. For example, for file `audit.20190415T151322.log.20190414T223342-2.enc` with keyring ID `audit-log-20190414T223342-2`: ```sql SELECT audit_log_encryption_password_get('audit-log-20190414T223342-2'); ``` -The return value of this function may look like the following: - ??? example "Expected output" ```{.text .no-copy} {"password":"{randomly-generated-alphanumeric-string}","iterations":568977} ``` + +## Additional reading + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — encryption and compression options + +* [Audit Log Filter security](audit-log-filter-security.md) + +* [Audit Log Filter file format overview](audit-log-filter-formats.md) + +* [Manage the Audit Log Filter files](manage-audit-log-filter.md) + +* [Keyring components and plugins overview](keyring-components-plugins-overview.md) + +* [Quickstart component keyring](quickstart-component-keyring.md) diff --git a/docs/audit-log-filter-definition-fields.md b/docs/audit-log-filter-definition-fields.md new file mode 100644 index 00000000000..8afb5a21a12 --- /dev/null +++ b/docs/audit-log-filter-definition-fields.md @@ -0,0 +1,238 @@ +# Audit Log Filter definition fields + +Canonical class, event, and field names for [`audit_log_filter_set_filter()`](audit-log-filter-variables.md#audit_log_filter_set_filterfilter_name-definition) validation. The names align with the Percona Server source [filter_definition_fields.md](https://github.com/percona/percona-server/blob/9.7/components/audit_log_filter/filter_definition_fields.md). + +For SQL command values (`create_db`, `select`, `grant`, and so on), typical event sequences for DDL/DCL/TCL operations, and the mapping between filter names and log output, see [Event classes, log output, and SQL commands](write-filter-definitions.md#event-classes-log-output-and-sql-commands). + +[`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode) decides which sections apply. The following notes contrast `REDUCED` and `FULL`. + +## Notes + +* Names on this page are filter-definition names. They may differ from JSON log output keys. + +* Field type reflects the current server validator (`get_event_field_value_type()`). + +* Some numeric-looking fields validate as `string` when the server leaves them untyped. + +* Only documented class names pass validation. + +* When [`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode) is `REDUCED` (the default), only the following events are tracked and accepted by filter-definition validation: + + * `general`: `status` + + * `connection`: `connect`, `disconnect`, `change_user` + + * `table_access`: `read`, `insert`, `update`, `delete` + + * `message`: `internal`, `user` + + In `REDUCED` mode, class names that exist only for extended auditing (`global_variable`, `command`, `query`, `stored_program`, `authentication`, and `parse`) are rejected entirely. Subclass names that are not in the preceding list (for example, `general/log` and `connection/pre_authenticate`) are also rejected during filter validation. At runtime, events outside the `REDUCED` set are silently skipped. With `FULL`, those six classes and their subclasses are valid filter targets. + +* Lifecycle-related records with class names `audit`, `server_startup`, and `server_shutdown` are not valid filter-definition targets. The audit log filter ignores startup and shutdown lifecycle events when they arrive. + +* For `connection.connection_type`, the validator accepts numeric values `0`–`5` and the pseudo-constants `::undefined`, `::tcp/ip`, `::socket`, `::named_pipe`, `::ssl`, and `::shared_memory`. + +## `general` + +Supported events: `log`, `error`, `result`, and `status`. + +REDUCED mode: only `status`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `general_error_code` | integer | Event error code. | +| `general_thread_id` | unsigned integer | Event thread ID. Aliased to `general_connection_id`. | +| `general_connection_id` | unsigned integer | Event connection ID. | +| `general_user.str` | string | User name recorded for the general event. | +| `general_user.length` | unsigned integer | User name length. | +| `general_command.str` | string | General command text, for example `Query`. | +| `general_command.length` | unsigned integer | General command text length. | +| `general_query.str` | string | SQL statement text associated with the event. | +| `general_query.length` | unsigned integer | SQL statement text length. | +| `general_host.str` | string | Client host name. | +| `general_host.length` | unsigned integer | Client host name length. | +| `general_sql_command.str` | string | SQL command name associated with the statement, for example `select`. | +| `general_sql_command.length` | unsigned integer | SQL command name length. | +| `general_external_user.str` | string | External user or OS login associated with the event. | +| `general_external_user.length` | unsigned integer | External user or OS login length. | +| `general_ip.str` | string | Client IP address. | +| `general_ip.length` | unsigned integer | Client IP address length. | + +## `connection` + +Supported events: `connect`, `disconnect`, `change_user`, and `pre_authenticate`. + +REDUCED mode: `connect`, `disconnect`, and `change_user`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `status` | integer | Current connection event status. | +| `connection_id` | unsigned integer | Connection ID. | +| `user.str` | string | User name of this connection. | +| `user.length` | unsigned integer | User name length. | +| `priv_user.str` | string | Privileged user name. | +| `priv_user.length` | unsigned integer | Privileged user name length. | +| `external_user.str` | string | External user name or OS login. | +| `external_user.length` | unsigned integer | External user name length. | +| `proxy_user.str` | string | Proxy user used for the connection. | +| `proxy_user.length` | unsigned integer | Proxy user name length. | +| `host.str` | string | Connection host name. | +| `host.length` | unsigned integer | Connection host name length. | +| `ip.str` | string | Connection IP address. | +| `ip.length` | unsigned integer | Connection IP address length. | +| `database.str` | string | Default database specified at connection time. | +| `database.length` | unsigned integer | Default database name length. | +| `connection_type` | integer | Connection type code: numeric `0`–`5` or pseudo-constant. See [Connection type constants](#connection-type-constants). | + +### Connection type constants + +`connection_type` values: + +| Value | Meaning | +| --- | --- | +| `0` or `::undefined` | Undefined | +| `1` or `::tcp/ip` | TCP/IP | +| `2` or `::socket` | Socket | +| `3` or `::named_pipe` | Named pipe | +| `4` or `::ssl` | TCP/IP with encryption | +| `5` or `::shared_memory` | Shared memory | + +## `table_access` + +Supported events: `read`, `insert`, `update`, and `delete`. + +REDUCED mode: all events. + +| Field name | Field type | Description | +| --- | --- | --- | +| `connection_id` | unsigned integer | Event connection ID. | +| `sql_command_id` | integer | SQL command ID. | +| `query.str` | string | SQL statement text. | +| `query.length` | unsigned integer | SQL statement text length. | +| `table_database.str` | string | Database name associated with event. | +| `table_database.length` | unsigned integer | Database name length. | +| `table_name.str` | string | Table name associated with event. | +| `table_name.length` | unsigned integer | Table name length. | + +## `global_variable` *(FULL mode only)* + +Supported events: `get` and `set`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `connection_id` | string | Event connection ID. | +| `variable_name.str` | string | Variable name. | +| `variable_name.length` | string | Variable name length. | +| `variable_value.str` | string | Variable value. | +| `variable_value.length` | string | Variable value length. | + +## `command` *(FULL mode only)* + +Supported events: `start` and `end`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `status` | string | Command event status code. | +| `connection_id` | string | Event connection ID. | +| `command.str` | string | Command text. | +| `command.length` | string | Command text length. | + +## `query` *(FULL mode only)* + +Supported events: `start`, `nested_start`, `status_end`, and `nested_status_end`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `status` | string | Query event status code. | +| `connection_id` | string | Event connection ID. | +| `sql_command_id` | string | SQL command string associated with the query event. The field name is retained as `sql_command_id` for compatibility. | +| `query.str` | string | SQL query text. | +| `query.length` | string | SQL query text length. | +| `query_charset` | string | SQL query character set name. | + +## `stored_program` *(FULL mode only)* + +Supported events: `execute`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `connection_id` | string | Event connection ID. | +| `database.str` | string | Database where the stored program is defined. | +| `database.length` | string | Database name length. | +| `name.str` | string | Stored program name. | +| `name.length` | string | Stored program name length. | + +## `authentication` *(FULL mode only)* + +Supported events: `flush`, `authid_create`, `credential_change`, `authid_rename`, and `authid_drop`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `status` | string | Authentication event status. | +| `connection_id` | string | Event connection ID. | +| `user.str` | string | User name. | +| `user.length` | string | User name length. | +| `host.str` | string | Host name. | +| `host.length` | string | Host name length. | + +## `message` + +Supported events: `internal` and `user`. + +REDUCED mode: all events. + +| Field name | Field type | Description | +| --- | --- | --- | +| `connection_id` | string | Event connection ID. | +| `component.str` | string | Component name. | +| `component.length` | string | Component name length. | +| `producer.str` | string | Message producer name. | +| `producer.length` | string | Message producer name length. | +| `message.str` | string | Message text. | +| `message.length` | string | Message text length. | + +## `parse` *(FULL mode only)* + +Supported events: `preparse` and `postparse`. + +| Field name | Field type | Description | +| --- | --- | --- | +| `connection_id` | string | Event connection ID. | +| `flags` | string | Parse rewrite flags value. | +| `query.str` | string | Original SQL query text. | +| `query.length` | string | Original SQL query text length. | +| `rewritten_query.str` | string | Rewritten SQL query text. | +| `rewritten_query.length` | string | Rewritten SQL query text length. | + +## Abortability + +Not every class or event supports execution blocking. The following table lists which filter-definition targets accept an `abort` item. When a filter tries to abort a non-abortable event, the server writes a warning to the error log and lets the statement through. See [Block statements with an audit log filter](block-statements-with-audit-filter.md). + +| Class | Event subclass | Can be aborted | +|---|---|---| +| `table_access` | `read`, `insert`, `update`, `delete` | Yes | +| `message` | `internal`, `user` | Yes | +| `query` | `start`, `status_end`, `nested_start`, `nested_status_end` | Yes (`FULL` mode only) | +| `stored_program` | `execute` | Yes (`FULL` mode only) | +| `connection` | all subclasses | No | +| `general` | all subclasses | No | +| `command` | all subclasses | No | +| `parse` | all subclasses | No | +| `global_variable` | all subclasses | No | +| `authentication` | all subclasses | No | + +## Additional reading + +* [Write audit_log_filter definitions](write-filter-definitions.md) + +* [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) + +* [Audit Log Filter restrictions](audit-log-filter-restrictions.md) + +* [Audit Log Filter file format overview](audit-log-filter-formats.md) + +* [Audit Log Filter overview](audit-log-filter-overview.md) + +* [Install the audit log filter](install-audit-log-filter.md) diff --git a/docs/audit-log-filter-formats.md b/docs/audit-log-filter-formats.md index 8a67ca57ed1..e05895c9088 100644 --- a/docs/audit-log-filter-formats.md +++ b/docs/audit-log-filter-formats.md @@ -1,24 +1,33 @@ # Audit Log Filter file format overview -When an auditable event occurs, the component writes a record to the log file. +On each auditable event, the component appends a record to the log. After startup, the first record describes the server and the startup options. Later records cover connections, disconnections, executed SQL, and other audit events. -After the component starts, the first record lists the description of the server and the options at startup. After the first record, the auditable events are connections, disconnections, SQL statements executed, and so on. Statements within stored procedures or triggers are not logged, only the top-level statements. +[`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode) controls which server actions become audit records. The file format does not change that selection. See the variable reference for `REDUCED` versus `FULL` and for releases before the variable existed. -If files are referenced by `LOAD_DATA`, the contents are not logged. +When `LOAD_DATA` references files, the component does not log file contents. -Set with the `audit_log_filter.format` system variable at startup. The available format types are the following; +Set the format with `audit_log_filter.format` at startup. The available formats follow: -| Format Type | Command | Description | +| Format type | Command | Description | |---|---|---| -| [XML (new style)](audit-log-filter-new.md) | `audit_log_filter.format=NEW` | The default format | -| [XML (old style)](audit-log-filter-old.md) | `audit_log_filter.format=OLD` | The original version of the XML format | -| [JSON](audit-log-filter-json.md) | `audit_log_filter.format=JSON` | Files written as a JSON array | +| [JSONL](audit-log-filter-json.md) | `audit_log_filter.format=JSONL` | Default. One compact JSON object per line inside a wrapping array. See the JSON and JSONL topic. | +| [JSON](audit-log-filter-json.md) | `audit_log_filter.format=JSON` | One top-level JSON array of events. | +| [XML (new style)](audit-log-filter-new.md) | `audit_log_filter.format=NEW` | XML layout. | -By default, the file contents in the new-style XML format are not compressed or encrypted. +By default, JSONL logs are neither compressed nor encrypted. -Changing the `audit_log_filter.format`, you should also change -the `audit_log_filter.file` name. For example, changing the `audit_log_filter.format` -to JSON, change the `audit_log_filter.file` to `audit.json`. If you don't change -the `audit_log_filter.file` name, then all audit log filter files have the same -base name and you won't be able to easily find when the format changed. +When you change `audit_log_filter.format`, rename `audit_log_filter.file` as well. For example, use `audit.json` or `audit.jsonl` for JSON or JSONL. Reusing one base name obscures format changes across rotated files. +## Additional reading + +* [Audit Log Filter overview](audit-log-filter-overview.md) + +* [Audit Log Filter format - JSON and JSONL](audit-log-filter-json.md) + +* [Audit Log Filter format - XML (new style)](audit-log-filter-new.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — `audit_log_filter.format` and `audit_log_filter.file` + +* [Reading Audit Log Filter files](reading-audit-log-filter-files.md) + +* [Audit Log Filter compression and encryption](audit-log-filter-compression-encryption.md) diff --git a/docs/audit-log-filter-json.md b/docs/audit-log-filter-json.md index d84931d19b4..66ad9174d1b 100644 --- a/docs/audit-log-filter-json.md +++ b/docs/audit-log-filter-json.md @@ -1,113 +1,158 @@ -# Audit Log Filter format - JSON +# Audit Log Filter format - JSON and JSONL -The JSON format has one top-level JSON array, which contain JSON objects with key-value pairs. These objects represent an event in the audit. Some pairs are listed in every audit record. The audit record type determines if other key-value pairs are listed. The order of the pairs within an audit record is not guaranteed. The value description may be truncated. +JSON and JSONL emit the same key-value pairs per event. Required keys appear in every record. Optional keys depend on event type and settings. Key order is not guaranteed, and long values may be truncated. -Certain statistics, such as query time and size, are only available in the JSON format and help detect activity outliers when analyzed. +Only the file layout differs: -```json +| Format | File structure | Set with | +|---|---|---| +| JSON | One top-level JSON array. Each event is a pretty-printed JSON object spanning multiple lines. | `audit_log_filter.format=JSON` | +| JSONL | One top-level JSON array. Each event is a single compact JSON object on its own line, separated by commas. | `audit_log_filter.format=JSONL` | + +Unlike plain [JSON Lines](https://jsonlines.org/), Percona JSONL keeps a wrapping JSON array and commas between lines. The file stays valid JSON and remains line-friendly for `grep`, `jq`, `wc -l`, streams, and aggregators. + +Compression and encryption behave like JSON. `audit_log_read()` and `audit_log_read_bookmark()` read both formats. + +JSON and JSONL alone expose some statistics such as query timing and size. Use those statistics to flag outliers in workload analysis. + +## Attributes + +Field sets match between JSON and JSONL. Only the wrapping differs. See the preceding table. + +Every event object includes at least: + +* `timestamp` + +* `id` + +* `class` + +* `event` + +Other common keys: + +| Name | Description | +|---|---| +| `account` | Database account for the event | +| `connection_data` | Client connection details; `connection_attributes` nest here on connection events. | +| `connection_id` | Client connection ID | +| `general_data` | Statement or command when `class` is `general` | +| `id` | Event ID | +| `login` | How the client attached to the server | +| `map` | Message payload. Message events also carry `account` and `login`. | +| `query_statistics` | Optional metrics for outlier detection | +| `shutdown_data` | Component shutdown | +| `startup_data` | Component startup; includes `server_id`, `os_version`, `mysql_version`, `args` | +| `table_access_data` | Table access details | +| `time` | UNIX timestamp (integer) when present | +| `timestamp` | UTC time `YYYY-MM-DD hh:mm:ss` | + +## JSON example + +The following shows four event types recorded in `REDUCED` event mode: startup, connection, table access, and general status. +```json [ { - "timestamp": "2023-03-29 11:17:03", + "timestamp": "2026-04-03 10:43:52", "id": 0, "class": "audit", - "server_id": 1 + "event": "startup", + "connection_id": 12, + "account": { "user": "root", "host": "localhost" }, + "login": { "user": "root", "os": "", "ip": "", "proxy": "" }, + "startup_data": { + "server_id": 1, + "os_version": "x86_64-Linux", + "mysql_version": "9.7.0-0", + "args": [ + "/usr/sbin/mysqld", + "--defaults-file=/etc/my.cnf", + "--basedir=/usr", + "--user=mysql", + "--datadir=/var/lib/mysql", + "--socket=/var/run/mysqld/mysqld.sock", + "--port=3306" + ] + } }, { - "timestamp": "2023-03-29 11:17:05", + "timestamp": "2026-04-03 10:43:53", "id": 1, - "class": "command", - "event": "command_start", - "connection_id": 1, - "command_data": { - "name": "command_start", + "class": "connection", + "event": "connect", + "connection_id": 39, + "account": { "user": "root", "host": "localhost" }, + "login": { "user": "root", "os": "", "ip": "", "proxy": "" }, + "connection_data": { + "connection_type": "socket", "status": 0, - "command": "query"} + "db": "test", + "connection_attributes": { + "_pid": "824388", + "_platform": "x86_64", + "_client_version": "8.0.45", + "_os": "Linux", + "_client_name": "libmysql" + } + } }, { - "timestamp": "2025-03-29 11:17:05", - "id": 332, - "class": "general", - "event": "log", - "connection_id": 11, - "account": { "user": "root[root] @ localhost []", "host": "localhost" }, - "login": { "user": "root[root] @ localhost []", "os": "", "ip": "", "proxy": "" }, - "general_data": { "status": 0 } + "timestamp": "2026-04-03 10:43:53", + "id": 9, + "class": "table_access", + "event": "read", + "connection_id": 40, + "account": { "user": "root", "host": "localhost" }, + "login": { "user": "root", "os": "", "ip": "", "proxy": "" }, + "table_access_data": { + "db": "test", + "table": "sbtest2", + "query": "SELECT c FROM sbtest2 WHERE id BETWEEN 83000 AND 83099", + "sql_command": "select" + } }, { - "timestamp": "2023-03-29 11:17:05", - "id": 3, - "class": "query", - "event": "query_start", - "connection_id": 11, - "query_data": { - "query": "CREATE TABLE t1 (c1 INT)", - "status": 0, - "sql_command": "create_table"} - }, - { - "timestamp": "2023-03-29 11:17:05", - "id": 4, - "class": "query", - "event": "query_status_end", - "connection_id": 11, - "query_data": { - "query": "CREATE TABLE t1 (c1 INT)", - "status": 0, - "sql_command": "create_table"} - }, - { - "timestamp": "2023-03-29 11:17:05", - "id": 5, + "timestamp": "2026-04-03 10:43:53", + "id": 11, "class": "general", "event": "status", - "connection_id": 11, - "account": { "user": "root[root] @ localhost []", "host": "localhost" }, - "login": { "user": "root[root] @ localhost []", "os": "", "ip": "", "proxy": "" }, + "connection_id": 40, + "account": { "user": "root", "host": "localhost" }, + "login": { "user": "root", "os": "", "ip": "", "proxy": "" }, "general_data": { "command": "Query", - "sql_command": "create_table", - "query": "CREATE TABLE t1 (c1 INT)", - "status": 0} - }, - { - "timestamp": "2023-03-29 11:17:05", - "id": 6, - "class": "command", - "event": "command_end", - "connection_id": 1, - "command_data": { - "name": "command_end", - "status": 0, - "command": "query"} + "sql_command": "select", + "query": "SELECT c FROM sbtest2 WHERE id BETWEEN 83000 AND 83099", + "status": 0 + } } ] ``` -The order of the attributes within the JSON object can vary. Certain attributes are in every element. Other attributes are optional and depend on the type of event and the filter settings or component settings. -The following fields are contained in each object: +## JSONL example -* `timestamp` -* `id` -* `class` -* `event` +In the JSONL format, each event is a single compact JSON object on its own line, separated by commas inside a wrapping JSON array. The same events from the preceding JSON example appear as follows: -The possible attributes in a JSON object are the following: +```json +[ +{"timestamp":"2026-04-03 10:43:52","id":0,"class":"audit","event":"startup","connection_id":12,"account":{"user":"root","host":"localhost"},"login":{"user":"root","os":"","ip":"","proxy":""},"startup_data":{"server_id":1,"os_version":"x86_64-Linux","mysql_version":"9.7.0-0","args":["/usr/sbin/mysqld","--defaults-file=/etc/my.cnf","--basedir=/usr","--user=mysql","--datadir=/var/lib/mysql","--socket=/var/run/mysqld/mysqld.sock","--port=3306"]}}, +{"timestamp":"2026-04-03 10:43:53","id":1,"class":"connection","event":"connect","connection_id":39,"account":{"user":"root","host":"localhost"},"login":{"user":"root","os":"","ip":"","proxy":""},"connection_data":{"connection_type":"socket","status":0,"db":"test","connection_attributes":{"_pid":"824388","_platform":"x86_64","_client_version":"8.0.45","_os":"Linux","_client_name":"libmysql"}}}, +{"timestamp":"2026-04-03 10:43:53","id":9,"class":"table_access","event":"read","connection_id":40,"account":{"user":"root","host":"localhost"},"login":{"user":"root","os":"","ip":"","proxy":""},"table_access_data":{"db":"test","table":"sbtest2","query":"SELECT c FROM sbtest2 WHERE id BETWEEN 83000 AND 83099","sql_command":"select"}}, +{"timestamp":"2026-04-03 10:43:53","id":11,"class":"general","event":"status","connection_id":40,"account":{"user":"root","host":"localhost"},"login":{"user":"root","os":"","ip":"","proxy":""},"general_data":{"command":"Query","sql_command":"select","query":"SELECT c FROM sbtest2 WHERE id BETWEEN 83000 AND 83099","status":0}} +] +``` -| Name | Description | -|---|---| -| `class` | Defines the type of event | -| `account` | Defines the MySQL account associated with the event. | -| `connection_data` | Defines the client connection. | -| `connection_id` | Defines the client connection identifier | -| `event` | Defines a subclass of the `event` class | -| `general_data` | Defines the executed statement or command when the audit record has a class value of `general`. | -| `id` | Defines the event ID | -| `login` | Defines how the client connected to the server | -| `query_statistics` | Defines optional query statistics and is used for outlier detection | -| `shutdown_data` | Defines the audit log filter termination | -| `startup_data` | Defines the initialization of the audit log filter component | -| `table_access_data` | Defines access to a table | -| `time` | Defines an integer that represents a UNIX timestamp | -| `timestamp` | Defines a UTC value in the `YYYY-MM_DD hh:mm:ss` format | +## Additional reading + +* [Audit Log Filter file format overview](audit-log-filter-formats.md) + +* [Audit Log Filter format - XML (new style)](audit-log-filter-new.md) + +* [Reading Audit Log Filter files](reading-audit-log-filter-files.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — `audit_log_read()`, `audit_log_read_bookmark()`, and format options + +* [Audit Log Filter compression and encryption](audit-log-filter-compression-encryption.md) +* [Manage the Audit Log Filter files](manage-audit-log-filter.md) diff --git a/docs/audit-log-filter-naming.md b/docs/audit-log-filter-naming.md index 471db845540..83a46037b84 100644 --- a/docs/audit-log-filter-naming.md +++ b/docs/audit-log-filter-naming.md @@ -2,38 +2,42 @@ ## Name qualities -The audit log filter file name has the following qualities: +An audit log path has: -* Optional directory name -* Base name -* Optional suffix +* Optional directory prefix -Using either [compression](audit-log-filter-compression-encryption.md) or [encryption](audit-log-filter-compression-encryption.md) adds the following suffixes: +* Base file name -* Compression adds the `.gz` suffix -* Encryption adds the `pwd_id.enc` suffix +* Optional suffix from compression or encryption -The `pwd_id` represents the password used for encrypting the log files. The audit log filter component stores passwords in the keyring. +[Compression](audit-log-filter-compression-encryption.md) and [encryption](audit-log-filter-compression-encryption.md) append suffixes: -You can combine compression and encryption, which adds both suffixes to the `audit_filter.log` name. +* Compression adds `.gz`. -The following table displays the possible ways a file can be named: +* Encryption adds `.pwd_id.enc`. -| Default name | Enabled feature | -| ----------------------- | ------------------------------ | -| audit.log | No compression or encryption | -| audit.log.gz | Compression | -| audit.log.pwd_id.enc | Encryption | -| audit.log.gz.pwd_id.enc | Compression, encryption | +`pwd_id` identifies the keyring entry for the password. The component stores keys in the keyring. + +With both features enabled, both suffixes appear, for example on `audit_filter.log`. + +Example names: + +| Default name | Enabled feature | +| ----------------------- | ---------------------------- | +| `audit_filter.log` | No compression or encryption | +| `audit_filter.log.gz` | Compression | +| `audit_filter.log.pwd_id.enc` | Encryption | +| `audit_filter.log.gz.pwd_id.enc` | Compression and encryption | ### Encryption ID format -The format for `pwd_id` is the following: +Each `pwd_id` contains: + +* UTC creation time as `YYYYMMDDThhmmss`. -* A UTC value in `YYYYMMDDThhmmss` format that represents when the password was created -* A sequence number that starts at `1` and increases if passwords have the same timestamp value +* A sequence that starts at `1` and increments when several passwords share one timestamp. -The following are examples of pwd_id values: +Examples: ```text 20230417T082215-1 @@ -41,7 +45,7 @@ The following are examples of pwd_id values: 20230301T061400-2 ``` -The following example is a list of the audit log filter files with the `pwd_id`: +Example encrypted file names: ```text audit_filter.log.20230417T082215-1.enc @@ -49,13 +53,33 @@ audit_filter.log.20230301T061400-1.enc audit_filter.log.20230301T061400-2.enc ``` -The current password has the largest sequence number. +The password with the highest sequence for a given timestamp is the current password. + +## Rotation sequence suffix + +Multiple rotations in the same second append `-N` so the server never overwrites a prior file: + +```text +audit_filter.20250401T120000.log -- first rotation at 12:00:00 +audit_filter.20250401T120000-1.log -- second rotation at 12:00:00 +``` + +Update parsers to accept the optional `-N` suffix. ## Renaming operations -During initialization, the component checks if a file with that name exists. -If it does, the component renames the file. The component writes to an empty file. +At startup, when the target path already has a file, the component renames the existing file and opens a new empty file. + +At shutdown, the component renames the active log file. + +## Additional reading + +* [Manage the Audit Log Filter files](manage-audit-log-filter.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — `audit_log_filter.file` -During termination, the component renames the file. +* [Audit Log Filter file format overview](audit-log-filter-formats.md) +* [Reading Audit Log Filter files](reading-audit-log-filter-files.md) +* [Audit Log Filter overview](audit-log-filter-overview.md) diff --git a/docs/audit-log-filter-new.md b/docs/audit-log-filter-new.md index 1c172e04a31..8a8c2445693 100644 --- a/docs/audit-log-filter-new.md +++ b/docs/audit-log-filter-new.md @@ -1,112 +1,425 @@ # Audit Log Filter format - XML (new style) -The filter writes the audit log filter file in XML. The XML file uses -UTF-8. +The NEW XML formatter emits: -The is the root element and this element contains -<AUDIT_RECORD> elements. Each <AUDIT_RECORD> element contains specific -information about an event that is audited. +* 1-based `` values, not 0-based. -For each new file, the Audit Log Filter component writes the XML -declaration and the root element tag. The component writes the closing - root element when closing the file. If the file is open, this -closing element is not available. +* Self-closing empty tags (``) instead of empty pairs. -```xml +* Single-space indent per level. + +* Stable element order for easier parsers and upgrades. + +* Rich startup records (`Audit`) that carry ``, ``, ``, and ``. + +* Message events with ``, ``, ``, and ``. Records also carry ``, ``, ``, ``, ``, and ``. + +* Consistent `` spellings (for example, `create_table` and `set_option`). + +* Event selection controlled by [`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode). `REDUCED` (default) keeps core classes. `FULL` adds lifecycle-heavy record types. See the variable page for the full lists. + +Code reference: `components/audit_log_filter/log_record_formatter/new.cc` and `base.cc`. + +With `audit_log_filter.format=NEW`, the component writes UTF-8 XML. + +The root `` element wraps `` elements, one per event. + +Element order matches the legacy audit plugin style. Parse by name, not by column position. + +Timestamps use the server local zone in `YYYY-MM-DDTHH:MM:SS` form without a ` UTC` suffix. +## Example — REDUCED mode (default) + +With `audit_log_filter.event_mode=REDUCED` (default), the formatter emits the primary record per action. The following sample mixes record types; actual fields depend on filters and configuration. + +```xml - - Audit - 0_2023-03-29T11:11:43 - 2023-03-29T11:11:43 - 1 - - - Command Start - 1_2023-03-29T11:11:45 - 2023-03-29T11:11:45 - 0 - 1 - query - - - Query - 2_2023-03-29T11:11:45 - 2023-03-29T11:11:45 - create_table - 11 - localhost - - root[root] @ localhost [] - - CREATE TABLE t1 (c1 INT) - 0 - - - Query Start - 3_2023-03-29T11:11:45 - 2023-03-29T11:11:45 - 0 - 11 - create_table - CREATE TABLE t1 (c1 INT) - - - Query - 4_2023-03-29T11:11:45 - 2023-03-29T11:11:45 - create_table - 11 - localhost - - root[root] @ localhost [] - - CREATE TABLE t1 (c1 INT) - 0 - - - Command End - 5_2023-03-29T11:11:45 - 2023-03-29T11:11:45 - 0 - 1 - query - + + 2026-03-27T20:00:59 + 1_2026-03-27T20:00:59 + Audit + 1 + 1 + /usr/sbin/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --port=3306 --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql + x86_64-Linux + 9.7.0-0 + + + 2026-03-27T20:00:59 + 2_2026-03-27T20:00:59 + Connect + 20 + 0 + 0 + root + + localhost + 127.0.0.1 + connect + TCP/IP + root + + + + + 2026-03-27T20:00:59 + 28_2026-03-27T20:00:59 + TableRead + 20 + root[root] @ localhost [127.0.0.1] + + localhost + 127.0.0.1 + select + SELECT * FROM t_access + test + t_access
+ + + 2026-03-27T20:00:59 + 29_2026-03-27T20:00:59 + Query + 20 + 0 + 0 + root[root] @ localhost [127.0.0.1] + + localhost + 127.0.0.1 + select + SELECT * FROM t_access + + + 2026-03-27T20:00:59 + 45_2026-03-27T20:00:59 + Message + 20 + 0 + 0 + root[root] @ localhost [127.0.0.1] + + localhost + 127.0.0.1 + internal + test_audit_api_message + test_audit_api_message + test_audit_api_message_internal + + + my_numeric_key + -9223372036854775808 + + + + + 2026-03-27T20:00:59 + 88_2026-03-27T20:00:59 + Connect + 22 + 0 + 0 + root + + localhost + 127.0.0.1 + connect + SSL + + + _pid + 764350 + + + _platform + x86_64 + + + _os + Linux + + + _client_name + libmysql + + + _client_version + 8.0.45 + + + program_name + mysqladmin + + + root + + + + + 2026-03-27T20:00:59 + 90_2026-03-27T20:00:59 + Quit + 22 + 0 + 0 + root + + localhost + 127.0.0.1 + connect + SSL + + + 2026-03-27T20:01:00 + 91_2026-03-27T20:01:00 + NoAudit + 1 + ``` -The order of the attributes within an <AUDIT_RECORD> can vary. Certain attributes are in every element. Other attributes are optional and depend on the type of audit record. - -The attributes in every element are the following: - -| Attribute Name | Description | -| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `` | The action that generated the audit record. | -| `` | The `` consists of a sequence number and a timestamp value. The sequence number is initialized when the component opens the audit log filter file. | -| `` | Displays the date and time when the audit event happened. | - -The optional attributes are the following: - -| Attribute Name | Description | -| ----------------------- | ---------------------- | -| `` | Contains the type of performed action. | -| `` | Contains the client connection identifier. | -| ``| Contains the client connection attributes. Each attribute has a `` and `` pair. | -| `` | Contains the type of connection security. | -| `` | Contains the database name. | -| `` | Contains the client's hostname. | -| `` | Contains the client's IP address. | -| `` | Contains the MySQL server version. | -| `` | Contains the user name used during an external authentication, for example, if the user is authenticated through an LDAP component. If the authentication component does not set a value or the user is authenticated using MySQL authentication, this value is empty. | -| `` | Contains the server's operating system. | -| `` | Contains the user name used by the server when checking privileges. This name may be different than ``. | -| `` | Contains the proxy user. If a proxy is not used, the value is empty. | -| `` | Contains the server ID. | -| `` | Contains the text of the SQL statement. | -| `` | Contains the startup options. These options may be provided by the command line or files. | -| `` | Contains the status of a command. A 0 (zero) is a success. A nonzero value is an error. | -| `` | Contains the status of a command, which either succeeds (0) or an error occurred (1). | -| `` | Contains the table name. | -| `` | Contains the user name from the client. This name may be different than ``. | -| `` | Contains the audit log filter format. | \ No newline at end of file +## Example — FULL mode (additional events) + +With `audit_log_filter.event_mode=FULL`, the formatter adds lifecycle records that wrap each action. The following fragment supplements the REDUCED example. + +```xml + + + 2026-04-03T13:19:21 + 2_2026-04-03T13:19:21 + Pre Authenticate + 20 + 0 + 0 + + + localhost + 127.0.0.1 + connect + TCP/IP + + + + + Command Start + 5_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + 0 + 20 + Query + + + + + Preparse + 6_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + Parse + 20 + 0 + INSERT INTO t_access VALUES (1, 'inserted') + + + + Postparse + 7_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + Parse + 20 + 0 + INSERT INTO t_access VALUES (1, 'inserted') + + + + + + Query Start + 124_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + 0 + 20 + insert + INSERT INTO t_access VALUES (1, 'inserted') + + + Query Status End + 126_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + 0 + 20 + insert + INSERT INTO t_access VALUES (1, 'inserted') + + + + Command End + 129_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + 0 + 20 + Query + + + + + Execute + 91_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + Stored Program + 20 + test + trigger_nested_query + + + + + Query Nested Start + 92_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + 0 + 20 + do + DO (SELECT 'nested query from stored procedure') + + + Query Nested Status End + 93_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + 0 + 20 + do + DO (SELECT 'nested query from stored procedure') + + + + + Variable Get + 284_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + select + 20 + sort_buffer_size + 262144 + + + Variable Set + 306_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + set_option + 20 + sort_buffer_size + 1048576 + + + + + Auth Credential Change + 337_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + Authentication + 20 + 0 + audit_tmp_user + localhost + + + Auth Authid Rename + 368_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + Authentication + 20 + 0 + audit_tmp_user + localhost + + + Auth Authid Drop + 399_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + Authentication + 20 + 0 + audit_renamed + localhost + + + Auth Flush + 430_2026-04-03T13:19:21 + 2026-04-03T13:19:21 + Authentication + 20 + 0 + + + +``` + +## Record descriptions + +| Record type | Required elements | Notes | +|---|---|---| +| Query (`Query`) | ``, ``, ``, `` (SQL command), usually ``, plus ``, ``, ``, and ``. | | +| Table access (`TableRead`, `TableInsert`, `TableUpdate`, `TableDelete`) | ``, `
`, ``, ``, ``, ``, ``, and ``. | No `` or ``. | +| Connection (`Connect`, `Quit`) | `` = `connect`, plus ``. | `Connect` adds ``, ``, and ``. | +| Client connection attributes | `` lists one `` element per attribute, each with `` and ``. | | +| Command Start and Command End (FULL) | `` and ``. `` holds the COM name. | Wraps each client command (`Query`, `Ping`, `Quit`). Host, user, and IP context are omitted. | +| Query Start / Query Status End and Query Nested Start / Query Nested Status End (FULL) | ``, ``, ``, and ``. | Brackets one SQL execution. Host, user, and IP context are omitted. | +| Preparse and Postparse (FULL) | `` = `Parse`, plus ``, ``, and ``. | Emitted before and after parse. | +| Execute (FULL) | `` = `Stored Program`, plus `` and ``. | Emitted for stored-program calls. | +| Variable Get and Variable Set (FULL) | ``, ``, and ``. | Emitted on global variable access. | +| Authentication subclasses (FULL) | `` = `Authentication` (or `connect` for pre-auth), plus ``, ``, and ``. | Subclasses: `Pre Authenticate`, `Auth Credential Change`, `Auth Authid Rename`, `Auth Authid Drop`, and `Auth Flush`. | + +Empty values use self-closing tags (``) for `USER`, `OS_LOGIN`, `HOST`, `IP`, `COMMAND_CLASS`, `PRIV_USER`, `PROXY_USER`, and `DB`. + +`` starts at 1. Indent one space per nesting level under ``. Records sit at one space, children at two, and so on. + +## Mandatory elements + +The following elements appear on every `` in the NEW XML format: + +| Element | Description | +| --- | --- | +| `` | Event subclass string. REDUCED mode: `Audit`, `Connect`, `Query`, `Ping`, `TableRead`, `TableInsert`, `TableUpdate`, `TableDelete`, `Message`, `Quit`, `NoAudit`. FULL mode adds: `Pre Authenticate`, `Command Start`, `Command End`, `Preparse`, `Postparse`, `Query Start`, `Query Status End`, `Query Nested Start`, `Query Nested Status End`, `Execute`, `Variable Get`, `Variable Set`, `Auth Credential Change`, `Auth Authid Rename`, `Auth Authid Drop`, `Auth Flush`. | +| `` | Sequence number and timestamp. See [`audit_log_filter` file handling](reading-audit-log-filter-files.md). The format is `SEQ_TIMESTAMP`, where the timestamp part matches the formatter's timestamp string. | +| `` | Local date and time for the event. | + +## Optional elements (by record category) + +Many elements appear only for specific event classes. The following table lists elements that the NEW XML formatter uses for at least one event type. The table does not promise that every field appears in every record. + +| Element | Description | +| --- | --- | +| `` | Meaning depends on the record: connection events (`Connect`, `Quit`) use `connect`; table-access and query events use the SQL command name (for example `select`, `insert`, `create_table`, `set_option`); message events use the message type (`internal`, `user`); command lifecycle events use the COM name (`Query`, `Ping`, `Quit`); parse events use `Parse`; stored-program events use `Stored Program`; authentication events use `Authentication`. Empty (``) on `Ping` records. | +| `` | Client connection ID. | +| `` | Nested `ATTRIBUTE` elements, each with `NAME` and `VALUE`. Omitted when no attributes exist. | +| `` | Connection security / transport (for example `TCP/IP`, `SSL`, `Socket`). | +| `` | Status code (`0` for success, non-zero for failure). Present on `Query`, `Connect`, `Quit`, `Ping`, `Message`, and authentication records. Also on `Command Start`/`End`, `Query Start`/`Status End` in FULL mode. Not present on table-access records. | +| `` | High-level status (`0` for success, `1` for failure). Present alongside `` on `Query`, `Connect`, `Quit`, `Ping`, and `Message` records. | +| `` | Statement or digest text when the event carries SQL text. | +| ``, ``, `` | Client context. Present on `Query`, `Connect`, `Quit`, `Ping`, table-access, `Message`, `Pre Authenticate`, and authentication records. Not emitted on `Command Start`/`End`, `Query Start`/`Status End`, `Preparse`/`Postparse`, `Execute`, or `Variable` records. | +| `` | External user from authentication (`external_user`). Present on `Query`, `Connect`, `Quit`, `Ping`, table-access, `Message`, and `Pre Authenticate` records. | +| ``, ``, `` | Included on `Connect` records. | +| `` | On `Audit` and `NoAudit` records. | +| `` | Audit log format version. On `Audit` (startup) records. | +| `` | Server command-line arguments as a single string. On `Audit` (startup) records. | +| `` | Operating system and architecture. On `Audit` (startup) records. | +| `` | Server version string. On `Audit` (startup) records. | +| ``, `
` | Database and table name on table-access records (`TableRead`, `TableInsert`, `TableUpdate`, `TableDelete`). `` also appears on `Execute` (stored program) records. | +| ``, `` | Global variable audit events (`Variable Get`, `Variable Set`). FULL mode only. | +| `` | Stored program name on `Execute` events (`` also appears). FULL mode only. | +| ``, `` | Parse events (`Preparse`, `Postparse`). `` also appears. FULL mode only. | +| ``, ``, ``, `` | Message events. `` contains `` children, each with a `` and ``. Message events also include ``, ``, ``, ``, ``, and `` fields. | + +The component XML-escapes characters such as `<`, `>`, `&`, and `"` in element text. Server-side limits may truncate very long values. + +## Additional reading + +* [Audit Log Filter file format overview](audit-log-filter-formats.md) + +* [Audit Log Filter format - JSON and JSONL](audit-log-filter-json.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — `audit_log_filter.event_mode` and `audit_log_filter.format` + +* [Reading Audit Log Filter files](reading-audit-log-filter-files.md) + +* [Audit Log Filter overview](audit-log-filter-overview.md) diff --git a/docs/audit-log-filter-old.md b/docs/audit-log-filter-old.md deleted file mode 100644 index 19e4f018bf4..00000000000 --- a/docs/audit-log-filter-old.md +++ /dev/null @@ -1,244 +0,0 @@ -# Audit Log Filter format - XML (old style) - -The old style XML format uses `` tag as the root element and adds the `` tag when the file closes. Each audited event is contained in an <AUDIT_RECORD> element. - -The order of the attributes within an <AUDIT_RECORD> can vary. Certain attributes are in every element. Other attributes are optional and depend on the type of audit record. - -```xml - - - - - - - - - - -``` - -The required attributes are the following: - - - - - HTML Table Generator - - - -
- - - - - - - - - - - - - - - - - - - - -
Attribute Name
Description
 NAME The action that generated the audit record.
 RECORD_ID The RECORD_ID consists of a sequence number and a timestamp value. The sequence number is initialized when the component opens the audit log filter file.
 TIMESTAMP Displays the date and time when the audit event happened.
- - - -The optional attributes are the following: - - - - - HTML Table Generator - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Attribute Name
Description
COMMAND_CLASS
Type of action performed
CONNECTION_IDClient connection identifier
CONNECTION_TYPEConnection security type
DBDatabase name
HOSTClient's hostname
IPClient's IP address
MYSQL_VERSIONServer version
OS_LOGINThe user name used during an external authentication, for example, if the user is authenticated through an LDAP component. If the authentication component does not set a value or the user is authenticated using MySQL authentication, this value is empty.
OS_VERSIONServer's operating system
PRIV_USERThe user name used by the server when checking privileges. This name may be different than USER.
PROXY_USERThe proxy user. If a proxy is not used, the value is empty.
SERVER_IDServer Identifier
SQLTEXTSQL statement text
STARTUP_OPTIONSServer startup options, either command line or config files
STATUSCommand's status - a 0 (zero) is a success, a non-zero is an error
STATUS_CODEA 0 (zero) is a success, a non-zero is an error
TABLETable name
USER Client's user name - this name may be different than PRIV_USER.
VERSIONFormat of audit log filter
- - \ No newline at end of file diff --git a/docs/audit-log-filter-overview.md b/docs/audit-log-filter-overview.md index 5deb0008529..2baac7cd9bc 100644 --- a/docs/audit-log-filter-overview.md +++ b/docs/audit-log-filter-overview.md @@ -1,40 +1,108 @@ # Audit Log Filter overview -The Audit Log Filter component allows you to monitor, log, and block a connection or query actively executed on the selected server. +The Audit Log Filter component records, audits, and can block matching connections or statements on the configured server. -Enabling the component produces a log file that contains a record of server activity. The log file has information on connections and databases accessed by that connection. +With the component enabled, the server writes an audit log file. The file captures connection events, executed statements, and the schemas that sessions touched. -The component uses the `mysql` system database to store filter and user account data. Set the [`audit_log_filter.database`](audit-log-filter-variables.md#audit_log_filterdatabase) variable at server startup to select a different database. +## Architecture -The `AUDIT_ADMIN` privilege is required to enable users to manage the Audit Log Filter component. +The following diagram shows how the Audit Log Filter component sits between the server core, the filter rules, and the audit output. + +![Audit Log Filter component architecture](_static/audit-log-filter-arch.png) + +### Why the filter component + +The component replaces the [legacy audit log plugin](audit-log-plugin.md). The design meets three goals that the plugin could not: + +* Change rules without restarting the server. + + Filter definitions and account assignments live in `mysql` system tables. Updates through [`audit_log_filter_set_filter()`](audit-log-filter-variables.md) and [`audit_log_filter_set_user()`](audit-log-filter-variables.md) take effect on new sessions and reloads. The server stays online. The legacy plugin required `audit_log_*` system-variable changes. Those changes often required a restart or plugin reinstall to adjust scope. + +* Scope audits per account, not globally. + + Each row in `mysql.audit_log_user` binds a user or user pattern to a named filter. The rows let you audit `admin@%` and `app@%` differently on the same instance. The legacy plugin offered only global include and exclude lists (`audit_log_include_accounts` and `audit_log_exclude_accounts`) on top of a single policy preset. + +* Express rules as data, not as knob settings. + + Filter definitions are JSON documents. A rule can match an event, a class, a subclass, or a specific field value. You combine matches with `and`, `or`, and `not`. Rules can call predefined variables and functions, block execution with `abort`, and redact statement text in place with `print` and `replace`. The legacy plugin had policy presets (`LOGINS`, `QUERIES`, `ALL`) with coarser control and no blocking or redaction. + +The design delivers a secondary benefit. Events that a filter drops never reach the formatter or the writer. The server therefore avoids the cost of auditing every event and discarding most of the data at the sink. Combined with per-account scoping, operators can raise audit detail on sensitive accounts without paying for verbose logging elsewhere. + +See [Write audit_log_filter definitions](write-filter-definitions.md) for the JSON grammar, [Block statements with an audit log filter](block-statements-with-audit-filter.md) for `abort`, and [Redact audit log fields](redact-audit-log-fields.md) for `print` and `replace`. + +## Audit data flow + +The following diagram traces audit events from the server through filtering and formatting into the log (file, syslog, or another handler). + +![Audit Log Filter data flow](_static/audit-data-flow.png) + +Set [`audit_log_filter.format`](audit-log-filter-variables.md#audit_log_filterformat) at startup. [Audit Log Filter file format overview](audit-log-filter-formats.md) compares JSONL (default), JSON, and NEW. + +[`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode) selects the event classes the component audits. The default is `REDUCED`. `FULL` enables every class the component supports, including classes beyond the four core classes. The variable reference lists the full class sets, validation rules, and behavior on older releases. + +The component stores filter definitions and account assignments in the `mysql` system database. Set [`audit_log_filter.database`](audit-log-filter-variables.md#audit_log_filterdatabase) at startup to use a different database. + +You need `AUDIT_ADMIN` to administer the Audit Log Filter component. ## Privileges -Define the privilege at runtime at the startup of the server. The associated Audit Log Filter privilege can be unavailable if the component is not enabled. +The server defines the Audit Log Filter privileges at startup. ### `AUDIT_ADMIN` -This privilege is defined by the server and enables the user to configure the component. +Grantees of `AUDIT_ADMIN` can configure the component. ### `AUDIT_ABORT_EXEMPT` -This privilege allows queries from a user account to always be executed. An `abort` item does not block them. This ability lets the user account regain access to a system if an audit is misconfigured. The query is logged due to the privilege. User accounts with the `SYSTEM_USER` privilege have the `AUDIT_ABORT_EXEMPT` privilege. +A filter may include an `abort` rule that blocks matching statements. To bypass those aborts, an account needs both `SYSTEM_USER` and the global `AUDIT_ABORT_EXEMPT` privilege granted on `*.*`. + +Only that pair lets matching statements run when a filter would otherwise abort the statement. The component still writes those statements to the audit log. + +## Audit log filter tables + +The Audit Log Filter component uses `mysql` system tables on `InnoDB`. The tables hold account assignments and filter JSON. Point the component at another database through `audit_log_filter.database` when the server starts. + +The `audit_log_filter` table stores named filter definitions: + +| Column name | Description | +|-------------|--------------------------------------------| +| NAME | Filter name | +| FILTER | JSON filter definition linked to the name | + +The `audit_log_user` table maps accounts to filters: + +| Column name | Description | +|--------------|--------------------------------| +| USER | MySQL account user part | +| HOST | MySQL account host part | +| FILTERNAME | Name of the assigned filter | + +### Filter storage hierarchy + +The following diagram shows how rows in `audit_log_filter` map to `audit_log_user`, including the default `%` account. + +![Audit Log Filter storage hierarchy](_static/audit-filter-hierarchy.png) + +On connect, the component loads one filter definition from the matching `USER`/`HOST` row in `mysql.audit_log_user`. If no row matches, the component falls back to the default account (`%`). A concrete account such as `admin`@`localhost` overrides a `%` assignment. + +Rules inside the filter's JSON apply after load. For example, `log` conditions that test `user.str` or `host.str` with `field` items narrow which events get written. The rules do not act as a second assigned filter. See [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md#using-the-audit-log-filter-functions) for assignment order and wildcards. See [Write audit_log_filter definitions](write-filter-definitions.md#test-event-field-values) for the JSON grammar that `log` conditions use. + +## Additional reading + +* [Install the audit log filter](install-audit-log-filter.md) + +* [Audit Log Filter quickstart](audit-log-filter-quickstart.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) -## Audit Log Filter tables +* [Write audit_log_filter definitions](write-filter-definitions.md) -The Audit Log Filter component uses `mysql` system database tables in the `InnoDB` storage engine. These tables store user account data and filter data. When you start the server, change the component's database with the `audit_log_filter.database` variable. +* [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md) -The `audit_log_filter` table stores the definitions of the filters and has the following column definitions: +* [Audit Log Filter file format overview](audit-log-filter-formats.md) -| Column name | Description | -|-------------|---------------------------------------------------------------| -| NAME | Name of the filter | -| FILTER | Definition of the filter linked to the name as a JSON value | +* [Migrate to the audit log filter component](migrate-to-audit-log-filter-component.md) — variable mapping, policy translation, worked example, and cutover for either audit plugin -The `audit_log_user` table stores account data and has the following column definitions: +* [Upgrade components](upgrade-components.md) — general plugin-to-component transition procedure -| Column name | Description | -|--------------|-------------------------------------| -| USER | The account name of the user | -| HOST | The account name of the host | -| FILTERNAME | The account filter name | +* [Audit log plugin](audit-log-plugin.md) — deprecated plugin reference diff --git a/docs/audit-log-filter-quickstart.md b/docs/audit-log-filter-quickstart.md new file mode 100644 index 00000000000..ad814a297ff --- /dev/null +++ b/docs/audit-log-filter-quickstart.md @@ -0,0 +1,57 @@ +# Audit Log Filter quickstart + +Use the following steps after you install the Audit Log Filter component and tables. See [install the audit log filter](install-audit-log-filter.md) when the component is not yet installed. + +You need `AUDIT_ADMIN`. Changing [`audit_log_filter.disable`](audit-log-filter-variables.md#audit_log_filterdisable) at runtime also requires `SYSTEM_VARIABLES_ADMIN`. + +!!! tip "Quickstart" + + 1. Optional: run `SHOW GLOBAL STATUS LIKE 'audit_log_filter_events_written';` and note the counter. + + 2. Run these five statements in order: + + ```sql + SET GLOBAL audit_log_filter.disable = false; + SELECT audit_log_filter_set_filter('log_all', '{ "filter": { "log": true } }'); + SELECT audit_log_filter_set_user('%', 'log_all'); + SELECT audit_log_filter_flush(); + SELECT 1; + ``` + + 3. Run `SHOW GLOBAL STATUS LIKE 'audit_log_filter_events_written';` again. The counter should increase. When the counter does not increase, verify that the component loaded, the tables exist, and [`audit_log_filter.disable`](audit-log-filter-variables.md#audit_log_filterdisable) is `false`. + + With the default `JSONL` [`audit_log_filter.format`](audit-log-filter-variables.md#audit_log_filterformat), read events with [`audit_log_read()`](audit-log-filter-variables.md#audit_log_read). See [Reading Audit Log Filter files](reading-audit-log-filter-files.md). You can also inspect the file named by [`audit_log_filter.file`](audit-log-filter-variables.md#audit_log_filterfile) on the server. The `JSON` format uses the same reader. The `NEW` (XML) format is read directly from the log file. + +## Next steps + +After the catch-all `log_all` rule works, replace the rule with a tighter JSON filter. Scope the filter by user, database, table, or event class. + +Call the filter UDFs to apply changes: + +* [`audit_log_filter_set_filter()`](audit-log-filter-variables.md#audit_log_filter_set_filterfilter_name-definition) + +* [`audit_log_filter_set_user()`](audit-log-filter-variables.md#audit_log_filter_set_useruser_name-filter_name) + +* [`audit_log_filter_flush()`](audit-log-filter-variables.md#audit_log_filter_flush) + +Author rules in [Write audit_log_filter definitions](write-filter-definitions.md). Compare assignment to JSON rules in [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md#assignment-vs-rules-inside-the-json). Validate names in [Audit Log Filter definition fields](audit-log-filter-definition-fields.md). + +When you switch to JSON or JSONL, read [Audit Log Filter file format overview](audit-log-filter-formats.md) and [Reading Audit Log Filter files](reading-audit-log-filter-files.md). + +For production, plan path, rotation, and retention with: + +* [`audit_log_filter.file`](audit-log-filter-variables.md#audit_log_filterfile) + +* [`audit_log_rotate()`](audit-log-filter-variables.md#audit_log_rotate) + +* [Manage the Audit Log Filter files](manage-audit-log-filter.md) + +## Additional reading + +* [Audit Log Filter overview](audit-log-filter-overview.md) + +* [Install the audit log filter](install-audit-log-filter.md) + +* [Disable Audit Log Filter logging](disable-audit-log-filter.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) diff --git a/docs/audit-log-filter-restrictions.md b/docs/audit-log-filter-restrictions.md index 014d44f5822..4284415cbaa 100644 --- a/docs/audit-log-filter-restrictions.md +++ b/docs/audit-log-filter-restrictions.md @@ -2,27 +2,69 @@ ## General restrictions -The Audit Log Filter has the following general restrictions: +* The component logs SQL statements only, not NoSQL APIs such as the Memcached interface. -The Audit Log Filter has the following general restrictions: +* Event coverage follows [`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode). In `REDUCED` mode, stored programs log the outer `CALL`, not each statement inside the body. The component never logs file contents referenced by statements such as `LOAD DATA`. -* Log only SQL statements. Statements made by NoSQL APIs, such as the - Memcached API, are not logged. +* Changing [`audit_log_filter.event_mode`](audit-log-filter-variables.md#audit_log_filterevent_mode) at runtime is not atomic across an event stream. Events from a single statement may split across the switch. For a clean cutover, pause the workload before changing the value or restart the server with the new value in the option file. See [Switching `event_mode` at runtime](audit-log-filter-variables.md#switching-event_mode-at-runtime). -* Log only the top-level statement. Statements within a stored procedure - or a trigger are not logged. Do not log the file contents for statements - like `LOAD_DATA`. +* On a cluster, install Audit Log Filter on every node that runs SQL. -* Require the component to be installed on each server used to execute SQL - on the cluster if used with a cluster. +* Aggregate audit data from every node yourself. The component does not centralize the data. -* Hold the application or user responsible for aggregating all the data from - each server used in the cluster if used with a cluster. +* Maintain a separate rule set per server. An uninitialized replica writes no audit rows for rules the replica lacks. Configure replication so that you do not blindly overwrite replica-local filter tables with source changes unless that outcome is intended. -* Each server must have its own audit log filter rules. If you do not set up the rules on the replica server, that server does not record the corresponding entries in the audit log. This design requires that the audit log configuration be performed separately for each server. + By default, replication can replicate `mysql.audit_log_filter` and `mysql.audit_log_user` from source to replica. Use replication filters or channels to exclude those tables when replicas should keep local rules. - As by default the content of the `mysql.audit_log_filter` and `mysql.audit_log_user` tables may be replicated from source to replica and may affect audit log rules created on the replica, it is recommended to configure replication in such a way that the changes in these tables are simply ignored. + Replicated table rows alone do not refresh in-memory filter state. Restart the server, reload the component, or run `audit_log_filter_flush()` to apply table changes to the running component. - Please notice that just changing the content of these tables (via replication channel) is not enough to automatically make changes to in-memory data structures in the `audit_log_filter` component that store information about active audit log filtering rules. However, this may happen after component reloading / server restart or manually calling `audit_log_filter_flush()`. +* Numeric fields in rules may use integer or string forms. `connection_type` accepts symbolic constants. See [`audit_log_filter_set_filter()`](audit-log-filter-variables.md#audit_log_filter_set_filterfilter_name-definition) and [Audit Log Filter definition fields](audit-log-filter-definition-fields.md). -* Filter only on string values. The audit log filter does not filter on integer values. All filter criteria must be specified as strings, even when the underlying value is numeric. For example, `connection_id` values must be specified as strings (for example, `"123"` rather than `123`), and status values must be specified as `"0"` or `"1"` rather than `0` or `1`. If you use integer values in your filter definition, you will see the error: `ERROR: Incorrect rule definition.` +## Synchronizing audit log filters between a source and a replica + +Replicate the filter tables and run `audit_log_filter_flush()` on the replica so replicated rows take effect in the component. Post-flush session behavior is documented under [`audit_log_filter_flush()`](audit-log-filter-variables.md#audit_log_filter_flush). + +### Procedure: scheduled event to flush filters on the replica + +1. Install Percona Server for MySQL on the source with Audit Log Filter. + +2. Clone a replica from a source backup. + +3. On the replica, create a one-minute event: + + ```sql + USE mysql; + CREATE EVENT auditflush + ON SCHEDULE EVERY 1 MINUTE + COMMENT 'Flush audit log filters every minute' + DO + SELECT audit_log_filter_flush(); + ``` + + The event applies replicated filter changes shortly after replication commits them. For session details after flush, see [`audit_log_filter_flush()`](audit-log-filter-variables.md#audit_log_filter_flush). + +4. On the source, define a filter, assign it to users (UDFs or direct table edits), and call `audit_log_filter_flush()` on the source when needed. + +5. After a minute, confirm the replica sees the filter; run queries as that user and verify the audit file. + +When nothing appears, wait for the next event tick or run `SELECT audit_log_filter_flush();` on the replica immediately. Confirm that replication applied the rows and that the event is `ENABLED`. Persistent gaps usually indicate a replication issue. Investigate the replica's replication health. + +When the component is missing on either host, install the component on source and replica independently. + +### Limitation + +Direct `INSERT`, `UPDATE`, or `DELETE` on the source without a source-side `audit_log_filter_flush()` still replicate to the replica. The scheduled event flushes on the replica. Replica filters update while the source in-memory state stays stale until someone flushes on the source. + +## Additional reading + +* [Audit Log Filter overview](audit-log-filter-overview.md) + +* [Filter the Audit Log Filter logs](filter-audit-log-filter-files.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — `audit_log_filter_flush()` + +* [Install the audit log filter](install-audit-log-filter.md) + +* [Upgrade Percona Server for MySQL](upgrade.md) + +* [Binlogging and replication improvements](binlogging-replication-improvements.md) diff --git a/docs/audit-log-filter-security.md b/docs/audit-log-filter-security.md index 24800aee287..703a2fac327 100644 --- a/docs/audit-log-filter-security.md +++ b/docs/audit-log-filter-security.md @@ -1,12 +1,29 @@ # Audit Log Filter security -The Audit Log Filter component generates audit log filter files. The directory -that contains these files should be accessible only to the following: +The Audit Log Filter component writes audit files. Restrict the log directory to trusted operators and ensure that the server can write to the directory. -* Users who must be able to view the log +Logs are plaintext by default and may hold credentials, SQL text, and other sensitive data. -* Server must be able to write to the directory +The default file under the data directory is `audit_filter.log`. Override the location with `audit_log_filter.file` at startup. -The files are not encrypted by default and may contain sensitive information. +When the parent directory is missing, the component errors and the server starts without Audit Log Filter active. -The default name for the file in the data directory is `audit_filter.log`. If needed, use the `audit_log_filter.file` system variable at server startup to change the location. Due to the log rotation, multiple audit log files may exist. +Rotation leaves multiple files on disk. Protect every generation. + +## Bypassing the OS page cache + +Audit log writes pass through the OS page cache by default. On busy servers with high audit throughput, recently written events can persist in kernel buffer memory until the page cache flushes them to disk. Other processes on the same host with read access to `/proc//pagemap` or `/dev/mem` may observe those buffers. + +Enable [`audit_log_filter.direct_io`](audit-log-filter-variables.md#audit_log_filterdirect_io) to open the audit log file with `O_DIRECT` and bypass the OS page cache. The component falls back to buffered I/O with a warning when the file system does not support `O_DIRECT` or when a direct write fails at runtime. The variable is a tech preview and requires a server restart to change. + +## Additional reading + +* [Audit Log Filter overview](audit-log-filter-overview.md) + +* [Audit Log Filter compression and encryption](audit-log-filter-compression-encryption.md) + +* [Manage the Audit Log Filter files](manage-audit-log-filter.md) + +* [Audit log filter functions, options, and variables](audit-log-filter-variables.md) — `audit_log_filter.file` and `audit_log_filter.handler` + +* [Install the audit log filter](install-audit-log-filter.md) diff --git a/docs/audit-log-filter-variables.md b/docs/audit-log-filter-variables.md index 90f70f89176..e427ef9195c 100644 --- a/docs/audit-log-filter-variables.md +++ b/docs/audit-log-filter-variables.md @@ -1,10 +1,10 @@ # Audit log filter functions, options, and variables -The following sections describe the [functions](#audit-log-filter-functions), [options, and variables](#audit-log-filter-options-and-variables) available in the audit log filter component. +Reference for audit log filter [functions](#audit-log-filter-functions) and [options / variables](#audit-log-filter-options-and-variables). ## Audit log filter functions -The following audit log filter functions are available. +Available UDFs: | Function name | | --- | @@ -22,15 +22,15 @@ The following audit log filter functions are available. ### audit_log_encryption_password_get(keyring_id) -This function returns the encryption password. Any keyring component or keyring component can be used, but the component or component must be enabled. If the component or component is not enabled, an error occurs. +Returns the audit encryption password (and iteration metadata) from the enabled keyring. Without a working keyring, the call errors. #### Parameters -`keyring_id` - If the function does not contain a keyring_id, the function returns the current encryption password. You can also request a specific encryption password with the keyring ID of either the current password or an archived password. +`keyring_id` — Omit to fetch the active password. Pass a keyring ID to read a specific archived or current entry. #### Returns -This function returns a JSON object containing the password, iterations count used by the password. +JSON with `password` and `iterations` for the requested keyring entry. #### Example @@ -50,27 +50,27 @@ SELECT audit_log_encryption_password_get(); ### audit_log_encryption_password_set(new_password) -This function sets the encryption password and stores the new password in the keyring. +Sets a new audit encryption password in the keyring. The call may also rotate the log file when encryption is active. See [Audit Log Filter compression and encryption](audit-log-filter-compression-encryption.md). #### Parameters -`password` - the password as a string. The maximum length is 766 bytes. +`new_password` — String up to 766 bytes. #### Returns -This function returns a string. An `OK` indicates a success. `ERROR` indicates a failure. +`OK` on success; an error string on failure. #### Example ```sql -SELECT audit_log_encryption_password_set(passw0rd); +SELECT audit_log_encryption_password_set('passw0rd'); ``` ??? example "Expected output" ```{.text .no-copy} +-----------------------------------------------------+ - | audit_log_encryption_password_set(passw0rd) | + | audit_log_encryption_password_set('passw0rd') | +-----------------------------------------------------+ | OK | +-----------------------------------------------------+ @@ -78,11 +78,11 @@ SELECT audit_log_encryption_password_set(passw0rd); ### audit_log_filter_flush() -This function updates the audit log filter tables and makes any changes operational. +Reloads filter JSON and account rows from `mysql.audit_log_filter` / `mysql.audit_log_user` into the component so memory matches disk. -Modifying the audit log filter tables directly with `INSERT`, `UPDATE`, or `DELETE` does not implement the modifications immediately. The tables must be flushed to have those changes take effect. +Table edits alone do not refresh every open session. Table edits include direct DML and [`audit_log_filter_set_filter()`](#audit_log_filter_set_filterfilter_name-definition). Call `audit_log_filter_flush()` when all sessions must see new rules. For details, see Persistence and refreshing on [`audit_log_filter_set_filter()`](#audit_log_filter_set_filterfilter_name-definition). -This function forces reloading all filters and should only be used if someone has modified the tables directly. +A flush detaches existing sessions until those sessions reconnect or run `CHANGE_USER`. New connections apply the reloaded registry immediately. When you cannot tolerate a gap, reconnect clients after flushing. #### Parameters @@ -110,23 +110,42 @@ SELECT audit_log_filter_flush(); ### audit_log_read() -If the audit log filter format is JSON, this function reads the audit log and returns an array of the audit events as a JSON string. Generates an error if the format is not JSON. +Reads JSON or JSONL audit files and returns events as a JSON array string. Other formats error. #### Parameters -None. If the start position is not provided, the read continues from the current position. +The function accepts a single JSON argument that selects a starting point. Four forms are valid: -Optional: You can specify a starting position for the read with `start` or a `timestamp` and an `id`, both items are considered a bookmark and can be used to identify an event. You must include both (`timestamp` and `id`) or an error is generated. If the `timestamp` does not include a `time` section, the function assumes the time is `00:00`. +* **Empty or omitted argument.** The function continues from the current read cursor in this session. Calling `audit_log_read()` with no argument is equivalent to passing `'{}'`. Both forms require a read context that an earlier call already established. Without a context, the call returns the `Wrong argument format` error. -You can also provide a `max_array_length` to limit the number of log events. +* **`start` envelope.** The function begins a new read sequence at the position described by an inner bookmark. The bookmark accepts `timestamp` and `id`. A `timestamp`-only start is legal. When the timestamp has no time part, the component assumes `00:00:00`. -Call[`audit_log_read_bookmark()`](#audit_log_read_bookmark) to return the most recently written event. +* **Bookmark literal.** Pass the JSON returned by [`audit_log_read_bookmark()`](#audit_log_read_bookmark) directly, with no `start` envelope. The bookmark form requires both `timestamp` and `id`. Passing one without the other returns the `Wrong argument format` error. Passing a non-string `timestamp` or a non-integer `id` returns the `bad bookmark format` error. + +* **JSON `null`.** The argument `'null'` closes the active read sequence in this session. Use this form to release the reader cursor before opening a new sequence with a different `start` or bookmark. The function returns the string `OK`. + +Position arguments are mutually exclusive. A single call cannot combine `start` with a top-level `timestamp` or `id`. A call cannot supply a new `start` or bookmark while a read context is already active in the session. To reposition, close the sequence with `'null'` and then issue a new `start` or bookmark literal. + +The optional `max_array_length` key caps how many events the call returns. Use it to page through dense periods or to limit response size. The key is valid in any form that supplies a position. + +The following table summarizes the top-level keys: + +| Key | Type | Purpose | +|---|---|---| +| `start` | object with `timestamp` | Start a new read sequence from the given UTC timestamp. | +| `timestamp` | string | Bookmark form. Must be paired with `id`. | +| `id` | unsigned integer | Bookmark form. Must be paired with `timestamp`. | +| `max_array_length` | unsigned integer | Cap on the number of events returned per call. Combines with either start form. | + +Seed reads with [`audit_log_read_bookmark()`](#audit_log_read_bookmark) when you want to resume from the tail of the log. #### Returns -This function returns a string of a JSON array of the audit events or a JSON NULL value. Returns `NULL` and generates an error if the call fails. +JSON array text, JSON `NULL` when no more events are available, the literal string `OK` when the call closes the active read sequence with the `'null'` argument, or an error. Output is strictly valid JSON, honors `max_array_length`, and resumes cleanly from bookmarks. -#### Example +#### Examples + +Resume from the most recent event (typical pattern): ```sql SELECT audit_log_read(audit_log_read_bookmark()); @@ -136,17 +155,53 @@ SELECT audit_log_read(audit_log_read_bookmark()); ```{.text .no-copy} +------------------------------------------------------------------------------+ - | audit_log_read(audit_log_read_bookmark()) | + | audit_log_read(audit_log_read_bookmark()) | +------------------------------------------------------------------------------+ - | [{"timestamp" : "2023-06-02 09:43:25", "id": 10,"class":"connection",] | + | [{"timestamp" : "2023-06-02 09:43:25", "id": 10, "class":"connection"}] | +------------------------------------------------------------------------------+ ``` +Start from a specific date and time: + +```sql +SELECT audit_log_read('{"start": {"timestamp": "2026-05-20 12:28:10"}}'); +``` + +Start from a date with no time portion (the component assumes `00:00:00`): + +```sql +SELECT audit_log_read('{"start": {"timestamp": "2026-05-20"}}'); +``` + +Cap the response size with `max_array_length`: + +```sql +SELECT audit_log_read('{"start": {"timestamp": "2026-05-20 12:28:10"}, "max_array_length": 3}'); +``` + +Resume from an explicit bookmark (both `timestamp` and `id` required, top-level, no `start` wrapper): + +```sql +SELECT audit_log_read('{"timestamp": "2026-05-20 12:28:10", "id": 1561422}'); +``` + +Continue the current read sequence (no argument, requires an active reader context): + +```sql +SELECT audit_log_read(); +``` + +Close the current read sequence: + +```sql +SELECT audit_log_read('null'); +``` + ### audit_log_read_bookmark() -This function provides a bookmark for the most recently written audit log event as a JSON string. Generates an error if the format is not JSON. +Returns a JSON bookmark for the latest event. Supported for JSON and JSONL formats only. Other formats return an error. -When this function is used with [`audit_log_read()`](#audit_log_read), the read starts reading at the specified position. +Pass the bookmark into [`audit_log_read()`](#audit_log_read) to begin there. ```sql SELECT audit_log_read(audit_log_read_bookmark()); @@ -178,15 +233,13 @@ SELECT audit_log_read_bookmark(); ### `audit_log_session_filter_id()` -This function returns the internal ID of the audit log filter in the current session. - -Returns 0 (zero) if the session has no assigned filter. +Returns the active filter ID for this session, or 0 when no filter applies. ### audit_log_filter_remove_filter(filter_name) -This function removes the selected filter from the current set of filters. +Drops a filter definition and clears `mysql.audit_log_user` rows that pointed at it. -If user accounts are assigned the selected filter, the user accounts are no longer filtered. The user accounts are removed from `audit_log_user`. If the user accounts are in a current session, they are detached from the selected filter and no longer logged. +Only sessions using that filter detach; others keep logging. #### Parameters @@ -217,11 +270,11 @@ SELECT audit_log_filter_remove_filter('filter-name'); ### audit_log_filter_remove_user(user_name) -This function removes the assignment of a filter from the selected user account. +Removes the `mysql.audit_log_user` row for that account pattern. -If the user account is in a current session, they are not affected. New sessions for this user account use the default account filter or are not logged. +Open sessions keep the cached filter until reconnect or `CHANGE_USER`. New sessions fall back to the `%` default, or stop auditing when no default exists. -If the user name is `%`, the default account filter is removed. +Passing `user_name = '%'` clears the default assignment. #### Parameters @@ -253,6 +306,10 @@ SELECT audit_log_filter_remove_user('user-name@localhost'); ### audit_log_rotate() +Rotates the active audit file and returns the archived name. + +Colliding timestamps add a `-N` suffix so rotations never overwrite each other. + #### Parameters None. @@ -269,9 +326,21 @@ SELECT audit_log_rotate(); ### audit_log_filter_set_filter(filter_name, definition) -This function, when provided with a filter name and definition, adds the filter. +Writes JSON for `filter_name` to `mysql.audit_log_filter` (create or update). Each stored revision gets a new filter ID. + +Validation runs at parse time. Bad fields, unknown classes or subclasses, empty arrays, stray JSON keys, and broken `print` rules abort the call with a detailed error. For example: + +``` +ERROR: Incorrect rule definition: Unknown field name "WRONG.str" for class "general" +``` + +In REDUCED event mode, a filter definition that references disabled event classes or subclasses fails the same way (error returned, filter not stored). + +#### Persistence and refreshing -The new filter has a different filter ID. Generates an error if the filter name exists. +`audit_log_filter_set_filter()` persists JSON but does not patch in-memory state for every open session. Run [`audit_log_filter_flush()`](#audit_log_filter_flush) to reload all definitions into the component. + +After a flush, existing sessions detach until reconnect or `CHANGE_USER`. New sessions apply changes immediately. #### Parameters @@ -287,7 +356,7 @@ This function returns either an `OK` for success or an error message for failure #### Example ```sql -SET @filter = '{ "filter_name": { "log": true }}' +SET @filter = '{ "filter": { "log": true } }'; SELECT audit_log_filter_set_filter('filter-name', @filter); ``` @@ -303,13 +372,19 @@ SELECT audit_log_filter_set_filter('filter-name', @filter); ### audit_log_filter_set_user(user_name, filter_name) -This function assigns the filter to the selected user account. +Binds `filter_name` to a login pattern in `mysql.audit_log_user`. + +Host wildcards (`%`, `_`) work in the host portion (`'usr1@%'`, `'usr2%172.16.10.%'`, `'usr3@%.mycorp.com'`, …). -The audit_log_filter_set_user() UDF accepts account names with wildcard characters (`'%'` and `'_'`) in the host part. For example, you can use `‘usr1@%'`, `‘usr2%192.168.0.%’`, or `'usr3@%.mycorp.com'`. +This UDF controls which named filter loads for a session. JSON `user` and `host` keys inside the filter still narrow events after load, but those keys are not a second assignment row. See [Assignment vs rules inside the JSON](filter-audit-log-filter-files.md#assignment-vs-rules-inside-the-json). -A user account can only have one filter. If the user account already has a filter, this function replaces the current filter. If the user account is in a current session, nothing happens. When the user account connects again the new filter is used. +Avoid overlapping patterns unless you accept ambiguous matches. Prefer literals, or one pattern plus `%`. -The user name, `%`, is the default account. The filter assigned to `%` is used by any user account without a defined filter. +One active mapping exists per account row. The call replaces any previous mapping. + +Open sessions keep the old mapping until reconnect or `CHANGE_USER`. Flush when you must realign all sessions immediately. + +The special user `%` is the default row used when no literal match exists. Specific `user@host` rows always beat `%`. #### Parameters @@ -339,13 +414,25 @@ SELECT audit_log_filter_set_user('user-name@localhost', 'filter-name'); ## Audit log filter options and variables +Audit Log Filter component uses the SQL form `audit_log_filter.