Skip to content

RaftBuildProcess

Rob Dobson edited this page Nov 7, 2026 · 4 revisions

Raft Build Process

This page describes the multi-stage build pipeline that compiles a Raft application for the ESP32 family of microcontrollers.

The Raft build system sits on top of the standard Espressif ESP IDF CMake build and extends it with SysType-based configuration, automatic library fetching, JSON-to-header generation and file system image creation. The build is normally invoked through RaftCLI but can also be driven directly with idf.py.

Overview

The build proceeds through four major layers:

  1. RaftCLI — resolves the SysType and the build variant, chooses the build method (Docker or native ESP IDF) and invokes idf.py.
  2. CMakeLists.txt (project root) — downloads and includes RaftBootstrap.cmake from a pinned RaftCore release.
  3. RaftBootstrap.cmake — resolves the SysType folder, reads features.cmake, fetches all Raft libraries and includes Phase 2.
  4. RaftBootstrapPhase2.cmake — merges config files, generates headers, sets up the partition table, builds the WebUI, prepares the file system image and hands control to ESP IDF's own project.cmake.
  raft build -s MySysType
        │
        ▼
  ┌───────────────┐
  │   RaftCLI     │  SysType and build variant, Docker / native IDF selection
  │   (idf.py -B) │
  └───────┬───────┘
          ▼
  ┌───────────────────────┐
  │  CMakeLists.txt       │  Downloads RaftBootstrap.cmake from GitHub release
  └───────┬───────────────┘
          ▼
  ┌───────────────────────┐
  │  RaftBootstrap.cmake  │  SysType folder, features.cmake, library fetch
  └───────┬───────────────┘
          ▼
  ┌───────────────────────┐
  │  RaftBootstrapPhase2  │  Config merging, header generation, ESP IDF,
  │  .cmake               │  partitions, WebUI, FS image, compile_commands
  └───────────────────────┘

Stage 1 — RaftCLI

RaftCLI is a Rust command-line application (raft) that wraps the Espressif idf.py tool. It provides a consistent developer experience for scaffolding, building, flashing, monitoring and OTA update of Raft applications.

SysType resolution in RaftCLI

When raft build (or raft run) is invoked, RaftCLI determines which SysType to build using the following priority:

Priority Source Example
1 -s command-line flag raft build -s MySysType
2 Last built SysType recorded in build/raft.info Automatically remembered
3 First non-Common folder found in systypes/ Alphabetical order

Build method selection

RaftCLI supports two build methods:

  • Docker (default) — uses a Dockerfile in the project root (typically based on espressif/idf:<version>) to run the build inside a container. This requires no local ESP IDF installation.
  • Native ESP IDF — uses a locally installed ESP IDF. Activated with --no-docker, or automatically when running inside an ESP IDF shell.

RaftCLI takes the required ESP IDF version from --idf-version, otherwise ESP_IDF_VERSION in features.cmake, otherwise the project's Dockerfile and, for native builds, searches for a matching installation in standard locations (~/esp on Linux/macOS, C:\Espressif\frameworks on Windows) or a path supplied with -e.

Build variant

raft build (and raft run) builds the dev variant unless --release (the release variant) or --variant <name> is given. The variant selects extra sdkconfig layers - see Build variants. It isn't remembered between builds. RaftCLI records the variant each SysType was built as in build/raft.info and deletes the SysType's build folder when it changes. (RaftCLI versions after 1.32.0)

What RaftCLI executes

After resolving the SysType and build method, RaftCLI essentially runs:

RAFT_ESP_IDF_VERSION=<version> RAFT_BUILD_VARIANT=<variant> idf.py -B build/<SysTypeName> build

The two environment variables pass the required ESP IDF version (checked against the ESP IDF in use) and the build variant to the CMake stages below. For a Docker build they are passed into the container with -e.

ESP IDF's idf.py invokes CMake which processes the project's CMakeLists.txt, triggering the remaining build stages.

Build info persistence

After a successful build, RaftCLI writes a build/raft.info JSON file containing, for example:

{
    "last_built_systype": "MySysType",
    "last_build_method": "docker",
    "last_idf_path_explicit": false,
    "idf_versions": { "MySysType": "6.1.0" },
    "local_libs": { "MySysType": "RaftCore" },
    "build_variants": { "MySysType": "dev" }
}

This allows subsequent raft build invocations to reuse the same SysType and build method without specifying them again. For each SysType it also records the ESP IDF version, the libraries used from raftdevlibs and the build variant, and the build folder is deleted when any of them changes. (The serial port, baud rates and other settings of raft flash, raft run and raft monitor are saved in the same file.)


Stage 2 — CMakeLists.txt

The project root CMakeLists.txt is deliberately minimal. Its sole job is to download the RaftBootstrap.cmake script from a specific RaftCore release and hand off control:

# Raft Project
cmake_minimum_required(VERSION 3.16)

# Bootstrap using RaftBootstrap script
set(BOOTSTRAP_URL "https://github.com/robdobsn/RaftCore/releases/download/v1.37.1/RaftBootstrap.cmake")
file(DOWNLOAD ${BOOTSTRAP_URL} "${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")
include("${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")

# Define the project dependencies
project(${_systype_name} DEPENDS ${ADDED_PROJECT_DEPENDENCIES})

Key points:

  • The BOOTSTRAP_URL pins a specific version of RaftBootstrap.cmake. To upgrade the Raft build system, change the version tag in this URL.
  • The file is downloaded into CMAKE_BINARY_DIR (i.e. build/<SysType>/) so each build configuration gets its own copy.
  • After the include(), the variables _systype_name and ADDED_PROJECT_DEPENDENCIES have been populated by the bootstrap scripts and are passed to the project() call.

Stage 3 — RaftBootstrap.cmake

This script is the first Raft-specific build logic. It runs during the CMake configuration phase and handles SysType folder resolution, features.cmake inclusion and library fetching.

3.1 SysType folder resolution

The bootstrap script determines the SysType from the build directory name:

  • If CMake is invoked with -B build/MySysType, the SysType name is MySysType.
  • If CMake is invoked with -B build (no sub-folder), the script scans systypes/ for the first non-Common directory and uses that.

The script verifies that systypes/<SysTypeName>/ exists, then sets:

set(BUILD_CONFIG_DIR "${CMAKE_SOURCE_DIR}/systypes/${_systype_name}")
set(RAFT_BUILD_ARTIFACTS_FOLDER "${CMAKE_BINARY_DIR}/raft")

The RAFT_BUILD_ARTIFACTS_FOLDER (build/<SysType>/raft/) is the directory where all generated intermediate files are placed during the build.

3.2 features.cmake

The SysType's features.cmake is included next. This file defines the build parameters for the SysType:

include(${BUILD_CONFIG_DIR}/features.cmake)

A typical features.cmake sets:

Variable Purpose Example
IDF_TARGET Target Espressif chip (lowercase) "esp32s3"
SYSTEM_VERSION Firmware version string add_compile_definitions(SYSTEM_VERSION="1.2.0")
RAFT_COMPONENTS List of Raft libraries and their versions RaftSysMods@main
FS_TYPE File system type "littlefs" or "spiffs"
FS_IMAGE_PATH Path to FS image source (relative to SysType dir) "../Common/FSImage"
UI_SOURCE_PATH Path to WebUI source (relative to SysType dir) "WebUI"
WEB_UI_GEN_FLAGS Optional flags for WebUI generation "--nogzip"
DEV_TYPE_JSON_FILES Additional DevType JSON files Paths to JSON files
RAFT_SDKCONFIG_FRAGMENTS sdkconfig fragments to apply (see 4.1) wifi-fast-connect deep-sleep-fast-wake
RAFT_BUILD_VARIANT Build variant for a build not run by the raft tool, e.g. idf.py run by hand (the raft tool always passes one) - see build variants. Default dev release

3.3 Raft library resolution

The script iterates over the RAFT_COMPONENTS list. Each entry has the format ComponentName@tag (or ComponentName#tag), where the tag is a Git branch, tag or commit.

For each component, the resolution order is:

  1. Local override — if <project-root>/raftdevlibs/<ComponentName> exists, use it directly. A prominent message is printed:
    ============================================================
      LOCAL (DEBUG) LIBRARY: RaftCore
      Using: /path/to/project/raftdevlibs/RaftCore
      (Skipping fetch from https://github.com/robdobsn/RaftCore.git)
    ============================================================
    
  2. Remote fetch — otherwise, the component is fetched from GitHub via CMake's FetchContent mechanism into build/<SysType>/raft/<ComponentName>/.

Full GitHub URLs are also supported in RAFT_COMPONENTS for components not hosted under the robdobsn organisation.

Each resolved component directory is added to EXTRA_COMPONENT_DIRS (making it visible to the ESP IDF component system) and to ADDED_PROJECT_DEPENDENCIES.

For details on local library overrides, see Local Raft Development Libraries.

3.4 Hand-off to Phase 2

After library resolution the bootstrap includes Phase 2 from the resolved RaftCore source directory:

include("${raftcore_SOURCE_DIR}/scripts/RaftBootstrapPhase2.cmake")

This ensures Phase 2 always comes from the specific version of RaftCore that the project depends on, not from the initially downloaded bootstrap.


Stage 4 — RaftBootstrapPhase2.cmake

Phase 2 performs the bulk of the build configuration. Each sub-stage adds custom targets that become project dependencies, ensuring correct ordering during the build.

4.1 SDKConfig merging

ESP IDF uses sdkconfig.defaults files to set default Kconfig values. The Raft build system applies these layers in order, later ones overriding earlier ones by CONFIG_ key:

  1. Common base: systypes/Common/sdkconfig.defaults
  2. Common build variant (optional): systypes/Common/sdkconfig.<variant>.defaults
  3. Fragments (optional): the sdkconfig fragments listed in RAFT_SDKCONFIG_FRAGMENTS, in the order listed
  4. SysType overrides: systypes/<SysType>/sdkconfig.defaults
  5. SysType build variant (optional): systypes/<SysType>/sdkconfig.<variant>.defaults

Merging is performed by MergeSysTypeConfigs.py --mode sdkconfig (with a --fragment argument per fragment and --common-variant / --systype-variant for the build variant's files). The merged result is written to build/<SysType>/raft/sdkconfig.defaults.merged and then used to seed the ESP IDF sdkconfig.

A custom target (sdkconfig) ensures the merged file is regenerated whenever any of the source files changes.

sdkconfig fragments

A fragment is a set of ESP-IDF settings for a purpose, selected in features.cmake (the SysType's or systypes/Common/features.cmake) instead of being copied into sdkconfig.defaults. (RaftCore versions after 1.64.0)

set(RAFT_SDKCONFIG_FRAGMENTS wifi-fast-connect deep-sleep-fast-wake)

Each entry is the name of a fragment in RaftCore's sdkconfig-fragments folder (the file <name>.defaults) or a path, relative to the project folder, to the project's own fragment (e.g. systypes/Common/low-power.defaults). A name which is neither stops the build with an error. As fragments come before the SysType's sdkconfig.defaults, a SysType can still override any setting a fragment makes. Without RAFT_SDKCONFIG_FRAGMENTS the merged output is the same as before.

Fragment Purpose
wifi-fast-connect Shorter DHCP on each WiFi connection: restores the last lease (CONFIG_LWIP_DHCP_RESTORE_LAST_IP) and doesn't ARP-check the offered address (CONFIG_LWIP_DHCP_DOES_NOT_CHECK_OFFERED_IP). Use with NetMan wifiSTAFastReconnect (see Fast reconnect)
deep-sleep-fast-wake Shorter boot after a wake from deep sleep: no app image check on a deep-sleep wake (CONFIG_BOOTLOADER_SKIP_VALIDATE_IN_DEEP_SLEEP, not used with secure boot) and bootloader warnings only (CONFIG_BOOTLOADER_LOG_LEVEL_WARN)

Build variants

A build variant selects extra sdkconfig layers so that one SysType can be built, for example, for development (logging on a UART that survives USB being unplugged, more checks) and for release. (RaftCore versions after 1.67.0)

raft build                    # the dev variant (the default)
raft build --release          # the release variant
raft build --variant <name>   # any other variant

For variant <name>, systypes/Common/sdkconfig.<name>.defaults is applied after Common/sdkconfig.defaults and systypes/<SysType>/sdkconfig.<name>.defaults after the SysType's sdkconfig.defaults (see the layers above). Both files are optional, so a project with no variant files builds the same whatever the variant. The usual layout puts what is shipped in sdkconfig.defaults and only the development differences in sdkconfig.dev.defaults.

When a variant file switches a Kconfig choice, it should turn the earlier choice off explicitly so the merged file sets only one of them, e.g. moving the console from USB Serial/JTAG to UART0:

# CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG is not set
CONFIG_ESP_CONSOLE_UART_CUSTOM=y
CONFIG_ESP_CONSOLE_UART_CUSTOM_NUM_0=y
CONFIG_ESP_CONSOLE_UART_TX_GPIO=43
CONFIG_ESP_CONSOLE_UART_RX_GPIO=44
CONFIG_ESP_CONSOLE_UART_BAUDRATE=921600
# CONFIG_ESP_CONSOLE_SECONDARY_NONE is not set
CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG=y

(ESP-IDF only uses CONFIG_ESP_CONSOLE_UART_BAUDRATE for a custom UART console - with CONFIG_ESP_CONSOLE_UART_DEFAULT it is fixed at 115200. GPIO43 / GPIO44 are UART0's pins on the esp32s3.) When the console is on a UART the SerialConsole SysMod sends console output through the UART driver's transmit buffer, so a task writing a log line doesn't wait while it is sent.

The raft tool always passes the variant in the RAFT_BUILD_VARIANT environment variable, which takes precedence over RAFT_BUILD_VARIANT set in features.cmake (that is only used by a build not run by the raft tool). It records it per SysType in raft.info, and deletes the SysType's build folder when the variant changes. The variant last built is written to build/<SysType>/raft/build_variant.txt; a build folder reconfigured with a different variant by other means (or made before build variants existed) has its sdkconfig regenerated.

4.2 SysTypes JSON merging

The SysType JSON configuration follows the same two-layer pattern:

  • Common base: systypes/Common/SysTypes.json
  • SysType overrides: systypes/<SysType>/SysTypes.json

Merging uses MergeSysTypeConfigs.py --mode systypes. This performs a shallow merge at the top-level key level — top-level keys from the SysType-specific file replace (not deep-merge) the same keys from the Common file.

The merged output is written to build/<SysType>/raft/SysTypes.json.merged.

4.3 SysType header generation

The merged SysTypes.json is converted into a C++ header (SysTypeInfoRecs.h) by GenerateSysTypes.py. This header contains the JSON configuration as C++ string literals that are compiled into the firmware.

The script supports two versioning mechanisms for hardware revisions:

  • HashHash versioning — keys suffixed with ##VersionName (e.g. "pin##RevA": 5, "pin##RevB": 12). Each version produces a separate compiled SysType record.
  • HwRev array versioning — values as arrays of {"__hwRevs__": [...], "__value__": ...} objects, allowing per-revision values for any JSON element.

A custom target (SysTypeInfoRecs) ensures the header is regenerated when the JSON inputs change.

4.4 DevType handling

Device type records describe I2C devices for automatic identification and polling by the DeviceManager. The build system processes DevTypes.json files into two generated C headers:

  • DeviceTypeRecords_generated.h — device type records, address lookup tables and scan priority lists.
  • DevicePollRecords_generated.h — poll result decode structures.

JSON sources are gathered from:

  1. Any paths specified in DEV_TYPE_JSON_FILES (set in features.cmake)
  2. systypes/Common/DevTypes.json (if it exists)
  3. systypes/<SysType>/DevTypes.json (if it exists)

The conversion is done by ProcessDevTypeJsonToC.py, which delegates to DecodeGenerator.py (C++ decode function and struct generation) and PseudocodeHandler.py (pseudocode-to-C++ transpilation). The generated headers are placed in build/<SysType>/raft/.

A custom target (generate_dev_ident_header) manages the dependency. The CMake DEPENDS clause tracks both the JSON source files and all three Python scripts, so changes to any of them trigger header regeneration.

4.5 ESP IDF integration

After all Raft-specific configuration is set up, the ESP IDF build system is included:

include($ENV{IDF_PATH}/tools/cmake/project.cmake)

At this point EXTRA_COMPONENT_DIRS contains all the fetched (or local) Raft library directories, and SDKCONFIG_DEFAULTS points to the merged sdkconfig chain. The standard ESP IDF component discovery, Kconfig processing and compilation proceed as normal.

4.6 Partition table

The partition table (partitions.csv) defines the flash memory layout. The build system selects the partition table using SysType-first-then-Common fallback:

  1. systypes/<SysType>/partitions.csv — used if present.
  2. systypes/Common/partitions.csv — fallback.

The selected file is copied to build/raft/partitions.csv (note: this is in the top-level build/raft/ directory, not per-SysType). This fixed location allows all SysType sdkconfig.defaults files to reference the same path:

CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="build_raft_artifacts/partitions.csv"

A custom target (partitions_csv) ensures the file is re-copied when the source changes.

4.7 WebUI build

If the SysType's features.cmake defines UI_SOURCE_PATH (a path relative to the SysType directory containing a web project with package.json), the build system compiles the WebUI:

  1. Validates that the source directory and package.json exist.
  2. Runs GenWebUI.py which executes npm install and npm run build in the WebUI source folder.
  3. The built files (HTML, JS, CSS) are gzip-compressed (unless --nogzip is specified via WEB_UI_GEN_FLAGS) and placed in build/<SysType>/raft/BuildWebUI/.

A custom target (WebUI) tracks the source files and rebuilds only when changes are detected.

4.8 FS image preparation

The file system image is assembled from two sources:

  1. FSImage folder — the static file system contents from <SysType dir>/<FS_IMAGE_PATH> (typically systypes/Common/FSImage). This is copied to build/<SysType>/raft/FSImage/.
  2. WebUI output — if a WebUI was built, the output files are copied on top of the FSImage directory.

A custom target (CopyFSAndWebUI) ensures the copying happens in the correct order (WebUI build first, then copy).

4.9 FS image generation

After the FS image folder is populated, the actual file system binary image is created by RaftGenFSImage.cmake. This is deferred until after ESP IDF components are loaded (so that partition information is available):

cmake_language(DEFER CALL include "${raftcore_SOURCE_DIR}/scripts/RaftGenFSImage.cmake")

The script supports two file system types:

  • LittleFS (default) — fetches the mklittlefs tool and creates a LittleFS partition image.
  • SPIFFS — uses the built-in ESP IDF SPIFFS image creation.

The image is sized to fit the fs partition defined in partitions.csv and is configured to be flashed alongside the firmware with idf.py flash.

4.10 compile_commands.json

As a final convenience step, the build copies compile_commands.json from the SysType build directory (build/<SysType>/compile_commands.json) up one level to the shared build/ directory. This allows VS Code's C/C++ IntelliSense (and other tools) to find the compile commands without needing to know the current SysType name:

add_custom_target(updateSharedCompileCommands ALL
    COMMAND ${CMAKE_COMMAND} -E copy_if_different
            "${CMAKE_BINARY_DIR}/compile_commands.json"
            "${CMAKE_BINARY_DIR}/.."
    DEPENDS "${CMAKE_BINARY_DIR}/compile_commands.json"
)

Intermediate build directories

build/raft.info

A JSON file written by RaftCLI after each successful build. Records the last-built SysType and build method so that subsequent builds can re-use the same settings without -s flags. See Stage 1 for the file format.

build/raft/

A top-level intermediate directory (not SysType-specific) containing:

File Purpose
partitions.csv Copy of the active partition table, placed at a fixed path so sdkconfig.defaults can reference it portably

build/<SysType>/raft/

The per-SysType intermediate artifacts directory (RAFT_BUILD_ARTIFACTS_FOLDER). Contains all generated files:

File / Folder Purpose
sdkconfig.defaults.merged Merged sdkconfig from Common + fragments + SysType (and the build variant's files)
build_variant.txt The build variant the sdkconfig was generated for
sdkconfig Final ESP IDF sdkconfig (copy of merged defaults)
SysTypes.json.merged Merged SysType JSON from Common + SysType
SysTypeInfoRecs.h Generated C++ header with SysType JSON as string literals
DeviceTypeRecords_generated.h Generated I2C device type records for DeviceManager
DevicePollRecords_generated.h Generated poll result decode structures
BuildWebUI/ Output of the WebUI npm build (gzipped HTML/JS/CSS)
FSImage/ Assembled file system contents (FSImage + WebUI merged)
RaftCore/ Fetched RaftCore source (if not using raftdevlibs)
RaftSysMods/ Fetched RaftSysMods source (if not using raftdevlibs)
(other libraries) Any other fetched Raft component sources

Build scripts reference

All build scripts live in the scripts/ directory of RaftCore:

Script Language Purpose
RaftBootstrap.cmake CMake SysType resolution, features.cmake, library fetch
RaftBootstrapPhase2.cmake CMake Config merging, header gen, ESP IDF integration, partitions, WebUI, FS image
MergeSysTypeConfigs.py Python Merges Common + SysType sdkconfig.defaults (with any sdkconfig fragments) and SysTypes.json files
GenerateSysTypes.py Python Converts merged SysTypes.json to SysTypeInfoRecs.h (C++ header with versioning)
ProcessDevTypeJsonToC.py Python Converts DevTypes.json to DeviceTypeRecords and DevicePollRecords headers
DecodeGenerator.py Python Generates C++ decode functions and output structs from device type definitions (called by ProcessDevTypeJsonToC.py)
PseudocodeHandler.py Python Transpiles pseudocode custom decode programs to C++ (called by DecodeGenerator.py)
GenWebUI.py Python Runs npm build on WebUI source, gzip-compresses output
RaftGenFSImage.cmake CMake Creates LittleFS or SPIFFS binary image from the assembled FSImage folder

Typical project structure

MyProject/
├── CMakeLists.txt              ← Stage 2: downloads RaftBootstrap
├── Dockerfile                  ← Docker build image (optional, for raft build)
├── components/
│   └── MySysMod/               ← User application SysMod(s)
├── main/                       ← ESP IDF main component
├── systypes/
│   ├── Common/
│   │   ├── SysTypes.json       ← Base configuration JSON
│   │   ├── sdkconfig.defaults  ← Base SDK config
│   │   ├── features.cmake      ← Build settings (IDF_TARGET, RAFT_COMPONENTS, etc.)
│   │   ├── partitions.csv      ← Flash partition table
│   │   ├── FSImage/            ← Static file system contents
│   │   ├── WebUI/              ← Web UI source (npm project)
│   │   └── DevTypes.json       ← Device type definitions (optional)
│   └── MySysType/
│       ├── SysTypes.json       ← SysType-specific overrides
│       ├── sdkconfig.defaults  ← SysType-specific SDK config overrides
│       └── features.cmake      ← Usually just includes Common/features.cmake
├── raftdevlibs/                ← Local library overrides (optional, gitignored)
│   └── RaftCore/               ← Local checkout for development
└── build/
    ├── raft.info               ← RaftCLI build state
    ├── raft/
    │   └── partitions.csv      ← Copied partition table
    └── MySysType/
        ├── raft/               ← Generated build artifacts
        │   ├── sdkconfig.defaults.merged
        │   ├── sdkconfig
        │   ├── SysTypes.json.merged
        │   ├── SysTypeInfoRecs.h
        │   ├── DeviceTypeRecords_generated.h
        │   ├── DevicePollRecords_generated.h
        │   ├── BuildWebUI/
        │   ├── FSImage/
        │   ├── RaftCore/       ← Fetched library source
        │   └── RaftSysMods/    ← Fetched library source
        ├── compile_commands.json
        └── ...                 ← ESP IDF build output

Clone this wiki locally