From 55a91922043911c63de93664dc40b477b4793136 Mon Sep 17 00:00:00 2001 From: David Cozens Date: Wed, 29 Jul 2026 09:07:35 +0100 Subject: [PATCH] feat: spool records to a file store with a CRC-16 at rest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A SolidSyslogBlockStore over a FileBlockDevice over the library's FatFs port, replacing the Null store. The service task drains the ring into the store and sends from there, so a failed send costs a retry rather than the record. Flash +11,576 B (+3,964 on the previous stage) RAM +7,092 B (+1,212) Log stack +448 B (unchanged) Service +736 B (+64) The log stack does not move: storing happens on the service task, and a task that calls Log still knows nothing about what happens after it returns. The service task's own high-water rises 64 bytes to 788, which the 2 KiB seam the buffered stage sized absorbs, so no RAM moves with it. The static RAM is pool allocation, not buffers. The 8192-byte block size is file capacity — nothing holds a block in memory, so the store costs its handles rather than its capacity. CRC-16 is a checksum, not tamper-evidence: it catches a truncated write or bit-rot, and anyone who can edit a stored record can recompute it. What it buys is knowing a record came back the way it went in, which is the prerequisite for spooling at all. One file per block, syslog00.log upward on the volume the device already mounts, four blocks, oldest discarded when full — a device that cannot reach its collector should keep the newest evidence rather than stop logging. SolidSyslog::FatFs is a header-configured upstream, so it is named in SOLIDSYSLOG_PLATFORMS and linked. The discard policy enumerator is SOLIDSYSLOG_DISCARD_POLICY_OLDEST. Co-Authored-By: Claude Opus 5 (1M context) --- CMakeLists.txt | 4 ++-- README.md | 12 +++++++----- app/syslog/Syslog.c | 22 +++++++++++++++++---- measurements/file-store.csv | 13 +++++++++++++ measurements/stages.tsv | 1 + run-report.md | 38 ++++++++++++++++++------------------- 6 files changed, 60 insertions(+), 30 deletions(-) create mode 100644 measurements/file-store.csv diff --git a/CMakeLists.txt b/CMakeLists.txt index de43e6f..060ef6f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -39,7 +39,7 @@ set(LWIP_CONTRIB_FREERTOS_DIR "${LWIP_DIR}/contrib/ports/freertos") # link target — only the header-configured packs below do. # https://docs.cososo.co.uk/solid-syslog/getting-started/#path-a--cmake-consumer # Pinned to a commit until there is a release tag to pin to. -set(SOLIDSYSLOG_PLATFORMS "LwipRaw;Atomics;FreeRtos" CACHE STRING "" FORCE) +set(SOLIDSYSLOG_PLATFORMS "LwipRaw;Atomics;FreeRtos;FatFs" CACHE STRING "" FORCE) # Compile-time limits. Every tunable is #ifndef-guarded, so this file only needs # the ones this device wants changed. @@ -171,7 +171,7 @@ target_include_directories(baseline PRIVATE # library, or context struct sizes diverge between consumer and library. target_compile_definitions(baseline PRIVATE MBEDTLS_USER_CONFIG_FILE=${MBEDTLS_USER_CONFIG_HEADER}) -target_link_libraries(baseline PRIVATE mbedtls mbedx509 mbedcrypto SolidSyslog SolidSyslog::LwipRaw SolidSyslog::FreeRtos) +target_link_libraries(baseline PRIVATE mbedtls mbedx509 mbedcrypto SolidSyslog SolidSyslog::LwipRaw SolidSyslog::FreeRtos SolidSyslog::FatFs) target_link_options(baseline PRIVATE -mcpu=cortex-m3 -mthumb diff --git a/README.md b/README.md index 834b954..54bce5f 100644 --- a/README.md +++ b/README.md @@ -10,15 +10,16 @@ It builds on a baseline that simulates the sort of device you might be adding th measures itself: see [docs/baseline.md](docs/baseline.md) for what the baseline is, how the figures are made, and how to run it. -## This stage — Time quality +## This stage — File store -The record gains a `timeQuality` SD-ELEMENT and a `sysUpTime` PARAM on `meta`. This device's clock -free-runs on the tick after one reading at boot, so it reports `isSynced="0"` and the collector -knows how far to trust the timestamp. +Records spool to a block store on the volume the device already mounts — four blocks, one file +each — instead of being held only in memory. The service task drains the ring into the store and +sends from there, so a failed send costs a retry rather than the record. The CRC-16 written with +each record catches corruption at rest, not tampering. -**Cost above baseline: Flash +7,612 B, RAM +5,880 B.** +**Cost above baseline: Flash +11,576 B, RAM +7,092 B.** @@ -42,6 +43,7 @@ committed as [`run-report.md`](run-report.md), and rewritten by every stage. | Buffered | logging that returns immediately, with the send moved off the logging task | +6,788 | +5,676 | | TCP | records the network retransmits instead of dropping, and a send that fails when the collector is gone | +7,320 | +5,856 | | Time quality | a timestamp the collector knows how far to trust, and an uptime that tells a reboot from a counter wrap | +7,612 | +5,880 | +| File store | records that survive a failed send, spooled to disk with a checksum at rest | +11,576 | +7,092 | *Deltas are bytes above the baseline, which is itself Flash 350,308 B, RAM 111,192 B.* diff --git a/app/syslog/Syslog.c b/app/syslog/Syslog.c index 3b7043f..7506d8e 100644 --- a/app/syslog/Syslog.c +++ b/app/syslog/Syslog.c @@ -9,10 +9,14 @@ #include "Syslog.h" +#include "SolidSyslogBlockStore.h" #include "SolidSyslogCircularBuffer.h" #include "SolidSyslogConfig.h" +#include "SolidSyslogCrc16Policy.h" #include "SolidSyslogEndpoint.h" #include "SolidSyslogEndpointHost.h" +#include "SolidSyslogFatFsFile.h" +#include "SolidSyslogFileBlockDevice.h" #include "SolidSyslogFreeRtosMutex.h" #include "SolidSyslogFreeRtosSysUpTime.h" #include "SolidSyslogLwipRawAddress.h" @@ -20,7 +24,6 @@ #include "SolidSyslogLwipRawResolver.h" #include "SolidSyslogLwipRawTcpStream.h" #include "SolidSyslogMetaSd.h" -#include "SolidSyslogNullStore.h" #include "SolidSyslogStdAtomicCounter.h" #include "SolidSyslogStreamSender.h" #include "SolidSyslogTimeQuality.h" @@ -46,6 +49,10 @@ * backlog the store is there to hold. */ #define SYSLOG_BUFFER_RECORDS 8U +/* One "NN.log" per block, on the volume the device already mounts. */ +#define SYSLOG_STORE_PREFIX "syslog" +#define SYSLOG_STORE_BLOCKS 4U + static struct SolidSyslog* s_logger = NULL; static uint8_t s_ring[SOLIDSYSLOG_CIRCULAR_BUFFER_RING_BYTES(SYSLOG_BUFFER_RECORDS)]; @@ -115,12 +122,19 @@ void Syslog_Start(void) s_sd[0] = SolidSyslogMetaSd_Create(&metaConfig); s_sd[1] = SolidSyslogTimeQualitySd_Create(SyslogTimeQuality); + /* Oldest discarded when the ceiling is reached: a device that cannot reach its + * collector should keep the newest evidence, not stop logging. */ + struct SolidSyslogBlockStoreConfig storeConfig = { + .BlockDevice = SolidSyslogFileBlockDevice_Create(SolidSyslogFatFsFile_Create(), SYSLOG_STORE_PREFIX, 0U), + .MaxBlocks = SYSLOG_STORE_BLOCKS, + .DiscardPolicy = SOLIDSYSLOG_DISCARD_POLICY_OLDEST, + .SecurityPolicy = SolidSyslogCrc16Policy_Create(), + }; + struct SolidSyslogConfig config = { .Buffer = SolidSyslogCircularBuffer_Create(SolidSyslogFreeRtosMutex_Create(), s_ring, sizeof(s_ring)), .Sender = sender, - /* No store-and-forward here. The Null object rather than NULL is how - * that is said out loud — NULL is reported as a fault. */ - .Store = SolidSyslogNullStore_Get(), + .Store = SolidSyslogBlockStore_Create(&storeConfig), /* PROCID stays unset — a bare-metal image has no process. */ .Clock = SyslogFields_Clock, .GetHostname = SyslogFields_Hostname, diff --git a/measurements/file-store.csv b/measurements/file-store.csv new file mode 100644 index 0000000..230c913 --- /dev/null +++ b/measurements/file-store.csv @@ -0,0 +1,13 @@ +# file-store figures (bytes) — captured by scripts/run.sh (CAPTURE=1). +# The device reads measurements/Baseline.csv as its frozen baseline and reports current-minus-Baseline. +flash_text,361252 +flash_data,632 +static_bss,117652 +heap_used,4440 +mbedtls_peak,21276 +mbedtls_free,11492 +lwip_mem_free,7576 +lwip_pbufs_free,14 +stack_log,568 +stack_service,788 +stack_harness,2848 diff --git a/measurements/stages.tsv b/measurements/stages.tsv index e2dc788..3a0e2db 100644 --- a/measurements/stages.tsv +++ b/measurements/stages.tsv @@ -20,3 +20,4 @@ message-cap Message cap a bounded record size, so a long message truncates inste buffered Buffered logging that returns immediately, with the send moved off the logging task tcp TCP records the network retransmits instead of dropping, and a send that fails when the collector is gone time-quality Time quality a timestamp the collector knows how far to trust, and an uptime that tells a reboot from a counter wrap +file-store File store records that survive a failed send, spooled to disk with a checksum at rest diff --git a/run-report.md b/run-report.md index 5156d1c..8d575dc 100644 --- a/run-report.md +++ b/run-report.md @@ -1,4 +1,4 @@ -# solid-syslog-example — run (time-quality) +# solid-syslog-example — run (file-store) ## Device (self-measured) @@ -10,16 +10,16 @@ [device] first record logged: yes [report] --- SolidSyslog cost above baseline (simulated existing application) --- [report] key,current,baseline,used_above_baseline -[report] flash_text,357424,349992,7432 -[report] flash_data,496,316,180 -[report] static_bss,116576,110876,5700 +[report] flash_text,361252,349992,11260 +[report] flash_data,632,316,316 +[report] static_bss,117652,110876,6776 [report] heap_used,4440,4440,0 -[report] mbedtls_peak,21352,21332,20 -[report] mbedtls_free,11416,11436,-20 +[report] mbedtls_peak,21276,21332,-56 +[report] mbedtls_free,11492,11436,56 [report] lwip_mem_free,7576,7576,0 -[report] lwip_pbufs_free,13,14,-1 +[report] lwip_pbufs_free,14,14,0 [report] stack_log,568,120,448 -[report] stack_service,724,52,672 +[report] stack_service,788,52,736 [report] stack_harness,2848,2840,8 [report] --- end --- [device] ready @@ -29,7 +29,7 @@ ```text text data bss dec hex filename - 357416 504 116576 474496 73d80 /w/build/baseline-cross/baseline.elf + 361244 640 117652 479536 75130 /w/build/baseline-cross/baseline.elf ``` ## Listeners (proved before the device ran) @@ -47,23 +47,23 @@ ## Collector (syslog-ng) received ```text -wire <134>1 2026-07-29T07:42:02.620000Z 10.0.2.15 solid-syslog-example - BOOT [meta sequenceId="1" sysUpTime="362"][timeQuality tzKnown="1" isSynced="0"] device started -parsed PRIORITY=134 TIMESTAMP=2026-07-29T07:42:02+00:00 HOSTNAME=10.0.2.15 APP_NAME=solid-syslog-example PROCID= MSGID=BOOT STRUCTURED_DATA=[meta sequenceId="1" sysUpTime="362"][timeQuality tzKnown="1" isSynced="0"] MSG=device started +wire <134>1 2026-07-29T08:06:15.410000Z 10.0.2.15 solid-syslog-example - BOOT [meta sequenceId="1" sysUpTime="241"][timeQuality tzKnown="1" isSynced="0"] device started +parsed PRIORITY=134 TIMESTAMP=2026-07-29T08:06:15+00:00 HOSTNAME=10.0.2.15 APP_NAME=solid-syslog-example PROCID= MSGID=BOOT STRUCTURED_DATA=[meta sequenceId="1" sysUpTime="241"][timeQuality tzKnown="1" isSynced="0"] MSG=device started ``` -## Self-check (vs measurements/time-quality.csv) +## Self-check (vs measurements/file-store.csv) ```text - OK flash_text: 357424 (expected 357424, Δ0) - OK flash_data: 496 (expected 496, Δ0) - OK static_bss: 116576 (expected 116576, Δ0) + OK flash_text: 361252 (expected 361252, Δ0) + OK flash_data: 632 (expected 632, Δ0) + OK static_bss: 117652 (expected 117652, Δ0) OK heap_used: 4440 (expected 4440, Δ0) - OK mbedtls_peak: 21352 (expected 21352, Δ0) - OK mbedtls_free: 11416 (expected 11416, Δ0) + OK mbedtls_peak: 21276 (expected 21276, Δ0) + OK mbedtls_free: 11492 (expected 11492, Δ0) OK lwip_mem_free: 7576 (expected 7576, Δ0) - OK lwip_pbufs_free: 13 (expected 13, Δ0) + OK lwip_pbufs_free: 14 (expected 14, Δ0) OK stack_log: 568 (expected 568, Δ0) - OK stack_service: 724 (expected 724, Δ0) + OK stack_service: 788 (expected 788, Δ0) OK stack_harness: 2848 (expected 2848, Δ0) ```