Repository navigation
RaftBuildProcess
This page describes the multi-stage build pipeline that compiles a Raft application for the ESP32 family of microcontrollers.
The Raft build system sits on top of the standard Espressif ESP IDF CMake build and extends it with SysType-based configuration, automatic library fetching, JSON-to-header generation and file system image creation. The build is normally invoked through RaftCLI but can also be driven directly with idf.py.
The build proceeds through four major layers:
-
RaftCLI — resolves the SysType and the build variant, chooses the build method (Docker or native ESP IDF) and invokes
idf.py. -
CMakeLists.txt(project root) — downloads and includesRaftBootstrap.cmakefrom a pinned RaftCore release. -
RaftBootstrap.cmake— resolves the SysType folder, readsfeatures.cmake, fetches all Raft libraries and includes Phase 2. -
RaftBootstrapPhase2.cmake— merges config files, generates headers, sets up the partition table, builds the WebUI, prepares the file system image and hands control to ESP IDF's ownproject.cmake.
raft build -s MySysType
│
▼
┌───────────────┐
│ RaftCLI │ SysType and build variant, Docker / native IDF selection
│ (idf.py -B) │
└───────┬───────┘
▼
┌───────────────────────┐
│ CMakeLists.txt │ Downloads RaftBootstrap.cmake from GitHub release
└───────┬───────────────┘
▼
┌───────────────────────┐
│ RaftBootstrap.cmake │ SysType folder, features.cmake, library fetch
└───────┬───────────────┘
▼
┌───────────────────────┐
│ RaftBootstrapPhase2 │ Config merging, header generation, ESP IDF,
│ .cmake │ partitions, WebUI, FS image, compile_commands
└───────────────────────┘
RaftCLI is a Rust command-line application (raft) that wraps the Espressif idf.py tool. It provides a consistent developer experience for scaffolding, building, flashing, monitoring and OTA update of Raft applications.
When raft build (or raft run) is invoked, RaftCLI determines which SysType to build using the following priority:
| Priority | Source | Example |
|---|---|---|
| 1 |
-s command-line flag |
raft build -s MySysType |
| 2 | Last built SysType recorded in build/raft.info
|
Automatically remembered |
| 3 | First non-Common folder found in systypes/
|
Alphabetical order |
RaftCLI supports two build methods:
-
Docker (default) — uses a
Dockerfilein the project root (typically based onespressif/idf:<version>) to run the build inside a container. This requires no local ESP IDF installation. -
Native ESP IDF — uses a locally installed ESP IDF. Activated with
--no-docker, or automatically when running inside an ESP IDF shell.
RaftCLI takes the required ESP IDF version from --idf-version, otherwise ESP_IDF_VERSION in features.cmake, otherwise the project's Dockerfile and, for native builds, searches for a matching installation in standard locations (~/esp on Linux/macOS, C:\Espressif\frameworks on Windows) or a path supplied with -e.
raft build (and raft run) builds the dev variant unless --release (the release variant) or --variant <name> is given. The variant selects extra sdkconfig layers - see Build variants. It isn't remembered between builds. RaftCLI records the variant each SysType was built as in build/raft.info and deletes the SysType's build folder when it changes. (RaftCLI versions after 1.32.0)
After resolving the SysType and build method, RaftCLI essentially runs:
RAFT_ESP_IDF_VERSION=<version> RAFT_BUILD_VARIANT=<variant> idf.py -B build/<SysTypeName> buildThe two environment variables pass the required ESP IDF version (checked against the ESP IDF in use) and the build variant to the CMake stages below. For a Docker build they are passed into the container with -e.
ESP IDF's idf.py invokes CMake which processes the project's CMakeLists.txt, triggering the remaining build stages.
After a successful build, RaftCLI writes a build/raft.info JSON file containing, for example:
{
"last_built_systype": "MySysType",
"last_build_method": "docker",
"last_idf_path_explicit": false,
"idf_versions": { "MySysType": "6.1.0" },
"local_libs": { "MySysType": "RaftCore" },
"build_variants": { "MySysType": "dev" }
}This allows subsequent raft build invocations to reuse the same SysType and build method without specifying them again. For each SysType it also records the ESP IDF version, the libraries used from raftdevlibs and the build variant, and the build folder is deleted when any of them changes. (The serial port, baud rates and other settings of raft flash, raft run and raft monitor are saved in the same file.)
The project root CMakeLists.txt is deliberately minimal. Its sole job is to download the RaftBootstrap.cmake script from a specific RaftCore release and hand off control:
# Raft Project
cmake_minimum_required(VERSION 3.16)
# Bootstrap using RaftBootstrap script
set(BOOTSTRAP_URL "https://github.com/robdobsn/RaftCore/releases/download/v1.37.1/RaftBootstrap.cmake")
file(DOWNLOAD ${BOOTSTRAP_URL} "${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")
include("${CMAKE_BINARY_DIR}/RaftBootstrap.cmake")
# Define the project dependencies
project(${_systype_name} DEPENDS ${ADDED_PROJECT_DEPENDENCIES})Key points:
- The
BOOTSTRAP_URLpins a specific version ofRaftBootstrap.cmake. To upgrade the Raft build system, change the version tag in this URL. - The file is downloaded into
CMAKE_BINARY_DIR(i.e.build/<SysType>/) so each build configuration gets its own copy. - After the
include(), the variables_systype_nameandADDED_PROJECT_DEPENDENCIEShave been populated by the bootstrap scripts and are passed to theproject()call.
This script is the first Raft-specific build logic. It runs during the CMake configuration phase and handles SysType folder resolution, features.cmake inclusion and library fetching.
The bootstrap script determines the SysType from the build directory name:
- If CMake is invoked with
-B build/MySysType, the SysType name isMySysType. - If CMake is invoked with
-B build(no sub-folder), the script scanssystypes/for the first non-Commondirectory and uses that.
The script verifies that systypes/<SysTypeName>/ exists, then sets:
set(BUILD_CONFIG_DIR "${CMAKE_SOURCE_DIR}/systypes/${_systype_name}")
set(RAFT_BUILD_ARTIFACTS_FOLDER "${CMAKE_BINARY_DIR}/raft")The RAFT_BUILD_ARTIFACTS_FOLDER (build/<SysType>/raft/) is the directory where all generated intermediate files are placed during the build.
The SysType's features.cmake is included next. This file defines the build parameters for the SysType:
include(${BUILD_CONFIG_DIR}/features.cmake)A typical features.cmake sets:
| Variable | Purpose | Example |
|---|---|---|
IDF_TARGET |
Target Espressif chip (lowercase) | "esp32s3" |
SYSTEM_VERSION |
Firmware version string | add_compile_definitions(SYSTEM_VERSION="1.2.0") |
RAFT_COMPONENTS |
List of Raft libraries and their versions | RaftSysMods@main |
FS_TYPE |
File system type |
"littlefs" or "spiffs"
|
FS_IMAGE_PATH |
Path to FS image source (relative to SysType dir) | "../Common/FSImage" |
UI_SOURCE_PATH |
Path to WebUI source (relative to SysType dir) | "WebUI" |
WEB_UI_GEN_FLAGS |
Optional flags for WebUI generation | "--nogzip" |
DEV_TYPE_JSON_FILES |
Additional DevType JSON files | Paths to JSON files |
RAFT_SDKCONFIG_FRAGMENTS |
sdkconfig fragments to apply (see 4.1) | wifi-fast-connect deep-sleep-fast-wake |
RAFT_BUILD_VARIANT |
Build variant for a build not run by the raft tool, e.g. idf.py run by hand (the raft tool always passes one) - see build variants. Default dev
|
release |
The script iterates over the RAFT_COMPONENTS list. Each entry has the format ComponentName@tag (or ComponentName#tag), where the tag is a Git branch, tag or commit.
For each component, the resolution order is:
-
Local override — if
<project-root>/raftdevlibs/<ComponentName>exists, use it directly. A prominent message is printed:============================================================ LOCAL (DEBUG) LIBRARY: RaftCore Using: /path/to/project/raftdevlibs/RaftCore (Skipping fetch from https://github.com/robdobsn/RaftCore.git) ============================================================ -
Remote fetch — otherwise, the component is fetched from GitHub via CMake's
FetchContentmechanism intobuild/<SysType>/raft/<ComponentName>/.
Full GitHub URLs are also supported in RAFT_COMPONENTS for components not hosted under the robdobsn organisation.
Each resolved component directory is added to EXTRA_COMPONENT_DIRS (making it visible to the ESP IDF component system) and to ADDED_PROJECT_DEPENDENCIES.
For details on local library overrides, see Local Raft Development Libraries.
After library resolution the bootstrap includes Phase 2 from the resolved RaftCore source directory:
include("${raftcore_SOURCE_DIR}/scripts/RaftBootstrapPhase2.cmake")This ensures Phase 2 always comes from the specific version of RaftCore that the project depends on, not from the initially downloaded bootstrap.
Phase 2 performs the bulk of the build configuration. Each sub-stage adds custom targets that become project dependencies, ensuring correct ordering during the build.
ESP IDF uses sdkconfig.defaults files to set default Kconfig values. The Raft build system applies these layers in order, later ones overriding earlier ones by CONFIG_ key:
-
Common base:
systypes/Common/sdkconfig.defaults -
Common build variant (optional):
systypes/Common/sdkconfig.<variant>.defaults -
Fragments (optional): the sdkconfig fragments listed in
RAFT_SDKCONFIG_FRAGMENTS, in the order listed -
SysType overrides:
systypes/<SysType>/sdkconfig.defaults -
SysType build variant (optional):
systypes/<SysType>/sdkconfig.<variant>.defaults
Merging is performed by MergeSysTypeConfigs.py --mode sdkconfig (with a --fragment argument per fragment and --common-variant / --systype-variant for the build variant's files). The merged result is written to build/<SysType>/raft/sdkconfig.defaults.merged and then used to seed the ESP IDF sdkconfig.
A custom target (sdkconfig) ensures the merged file is regenerated whenever any of the source files changes.
A fragment is a set of ESP-IDF settings for a purpose, selected in features.cmake (the SysType's or systypes/Common/features.cmake) instead of being copied into sdkconfig.defaults. (RaftCore versions after 1.64.0)
set(RAFT_SDKCONFIG_FRAGMENTS wifi-fast-connect deep-sleep-fast-wake)Each entry is the name of a fragment in RaftCore's sdkconfig-fragments folder (the file <name>.defaults) or a path, relative to the project folder, to the project's own fragment (e.g. systypes/Common/low-power.defaults). A name which is neither stops the build with an error. As fragments come before the SysType's sdkconfig.defaults, a SysType can still override any setting a fragment makes. Without RAFT_SDKCONFIG_FRAGMENTS the merged output is the same as before.
| Fragment | Purpose |
|---|---|
wifi-fast-connect |
Shorter DHCP on each WiFi connection: restores the last lease (CONFIG_LWIP_DHCP_RESTORE_LAST_IP) and doesn't ARP-check the offered address (CONFIG_LWIP_DHCP_DOES_NOT_CHECK_OFFERED_IP). Use with NetMan wifiSTAFastReconnect (see Fast reconnect) |
deep-sleep-fast-wake |
Shorter boot after a wake from deep sleep: no app image check on a deep-sleep wake (CONFIG_BOOTLOADER_SKIP_VALIDATE_IN_DEEP_SLEEP, not used with secure boot) and bootloader warnings only (CONFIG_BOOTLOADER_LOG_LEVEL_WARN) |
A build variant selects extra sdkconfig layers so that one SysType can be built, for example, for development (logging on a UART that survives USB being unplugged, more checks) and for release. (RaftCore versions after 1.67.0)
raft build # the dev variant (the default)
raft build --release # the release variant
raft build --variant <name> # any other variant
For variant <name>, systypes/Common/sdkconfig.<name>.defaults is applied after Common/sdkconfig.defaults and systypes/<SysType>/sdkconfig.<name>.defaults after the SysType's sdkconfig.defaults (see the layers above). Both files are optional, so a project with no variant files builds the same whatever the variant. The usual layout puts what is shipped in sdkconfig.defaults and only the development differences in sdkconfig.dev.defaults.
When a variant file switches a Kconfig choice, it should turn the earlier choice off explicitly so the merged file sets only one of them, e.g. moving the console from USB Serial/JTAG to UART0:
# CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG is not set
CONFIG_ESP_CONSOLE_UART_CUSTOM=y
CONFIG_ESP_CONSOLE_UART_CUSTOM_NUM_0=y
CONFIG_ESP_CONSOLE_UART_TX_GPIO=43
CONFIG_ESP_CONSOLE_UART_RX_GPIO=44
CONFIG_ESP_CONSOLE_UART_BAUDRATE=921600
# CONFIG_ESP_CONSOLE_SECONDARY_NONE is not set
CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG=y
(ESP-IDF only uses CONFIG_ESP_CONSOLE_UART_BAUDRATE for a custom UART console - with CONFIG_ESP_CONSOLE_UART_DEFAULT it is fixed at 115200. GPIO43 / GPIO44 are UART0's pins on the esp32s3.) When the console is on a UART the SerialConsole SysMod sends console output through the UART driver's transmit buffer, so a task writing a log line doesn't wait while it is sent.
The raft tool always passes the variant in the RAFT_BUILD_VARIANT environment variable, which takes precedence over RAFT_BUILD_VARIANT set in features.cmake (that is only used by a build not run by the raft tool). It records it per SysType in raft.info, and deletes the SysType's build folder when the variant changes. The variant last built is written to build/<SysType>/raft/build_variant.txt; a build folder reconfigured with a different variant by other means (or made before build variants existed) has its sdkconfig regenerated.
The SysType JSON configuration follows the same two-layer pattern:
-
Common base:
systypes/Common/SysTypes.json -
SysType overrides:
systypes/<SysType>/SysTypes.json
Merging uses MergeSysTypeConfigs.py --mode systypes. This performs a shallow merge at the top-level key level — top-level keys from the SysType-specific file replace (not deep-merge) the same keys from the Common file.
The merged output is written to build/<SysType>/raft/SysTypes.json.merged.
The merged SysTypes.json is converted into a C++ header (SysTypeInfoRecs.h) by GenerateSysTypes.py. This header contains the JSON configuration as C++ string literals that are compiled into the firmware.
The script supports two versioning mechanisms for hardware revisions:
-
HashHash versioning — keys suffixed with
##VersionName(e.g."pin##RevA": 5, "pin##RevB": 12). Each version produces a separate compiled SysType record. -
HwRev array versioning — values as arrays of
{"__hwRevs__": [...], "__value__": ...}objects, allowing per-revision values for any JSON element.
A custom target (SysTypeInfoRecs) ensures the header is regenerated when the JSON inputs change.
Device type records describe I2C devices for automatic identification and polling by the DeviceManager. The build system processes DevTypes.json files into two generated C headers:
-
DeviceTypeRecords_generated.h— device type records, address lookup tables and scan priority lists. -
DevicePollRecords_generated.h— poll result decode structures.
JSON sources are gathered from:
- Any paths specified in
DEV_TYPE_JSON_FILES(set infeatures.cmake) -
systypes/Common/DevTypes.json(if it exists) -
systypes/<SysType>/DevTypes.json(if it exists)
The conversion is done by ProcessDevTypeJsonToC.py, which delegates to DecodeGenerator.py (C++ decode function and struct generation) and PseudocodeHandler.py (pseudocode-to-C++ transpilation). The generated headers are placed in build/<SysType>/raft/.
A custom target (generate_dev_ident_header) manages the dependency. The CMake DEPENDS clause tracks both the JSON source files and all three Python scripts, so changes to any of them trigger header regeneration.
After all Raft-specific configuration is set up, the ESP IDF build system is included:
include($ENV{IDF_PATH}/tools/cmake/project.cmake)At this point EXTRA_COMPONENT_DIRS contains all the fetched (or local) Raft library directories, and SDKCONFIG_DEFAULTS points to the merged sdkconfig chain. The standard ESP IDF component discovery, Kconfig processing and compilation proceed as normal.
The partition table (partitions.csv) defines the flash memory layout. The build system selects the partition table using SysType-first-then-Common fallback:
-
systypes/<SysType>/partitions.csv— used if present. -
systypes/Common/partitions.csv— fallback.
The selected file is copied to build/raft/partitions.csv (note: this is in the top-level build/raft/ directory, not per-SysType). This fixed location allows all SysType sdkconfig.defaults files to reference the same path:
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="build_raft_artifacts/partitions.csv"A custom target (partitions_csv) ensures the file is re-copied when the source changes.
If the SysType's features.cmake defines UI_SOURCE_PATH (a path relative to the SysType directory containing a web project with package.json), the build system compiles the WebUI:
- Validates that the source directory and
package.jsonexist. - Runs
GenWebUI.pywhich executesnpm installandnpm run buildin the WebUI source folder. - The built files (HTML, JS, CSS) are gzip-compressed (unless
--nogzipis specified viaWEB_UI_GEN_FLAGS) and placed inbuild/<SysType>/raft/BuildWebUI/.
A custom target (WebUI) tracks the source files and rebuilds only when changes are detected.
The file system image is assembled from two sources:
-
FSImage folder — the static file system contents from
<SysType dir>/<FS_IMAGE_PATH>(typicallysystypes/Common/FSImage). This is copied tobuild/<SysType>/raft/FSImage/. - WebUI output — if a WebUI was built, the output files are copied on top of the FSImage directory.
A custom target (CopyFSAndWebUI) ensures the copying happens in the correct order (WebUI build first, then copy).
After the FS image folder is populated, the actual file system binary image is created by RaftGenFSImage.cmake. This is deferred until after ESP IDF components are loaded (so that partition information is available):
cmake_language(DEFER CALL include "${raftcore_SOURCE_DIR}/scripts/RaftGenFSImage.cmake")The script supports two file system types:
-
LittleFS (default) — fetches the
mklittlefstool and creates a LittleFS partition image. - SPIFFS — uses the built-in ESP IDF SPIFFS image creation.
The image is sized to fit the fs partition defined in partitions.csv and is configured to be flashed alongside the firmware with idf.py flash.
As a final convenience step, the build copies compile_commands.json from the SysType build directory (build/<SysType>/compile_commands.json) up one level to the shared build/ directory. This allows VS Code's C/C++ IntelliSense (and other tools) to find the compile commands without needing to know the current SysType name:
add_custom_target(updateSharedCompileCommands ALL
COMMAND ${CMAKE_COMMAND} -E copy_if_different
"${CMAKE_BINARY_DIR}/compile_commands.json"
"${CMAKE_BINARY_DIR}/.."
DEPENDS "${CMAKE_BINARY_DIR}/compile_commands.json"
)A JSON file written by RaftCLI after each successful build. Records the last-built SysType and build method so that subsequent builds can re-use the same settings without -s flags. See Stage 1 for the file format.
A top-level intermediate directory (not SysType-specific) containing:
| File | Purpose |
|---|---|
partitions.csv |
Copy of the active partition table, placed at a fixed path so sdkconfig.defaults can reference it portably |
The per-SysType intermediate artifacts directory (RAFT_BUILD_ARTIFACTS_FOLDER). Contains all generated files:
| File / Folder | Purpose |
|---|---|
sdkconfig.defaults.merged |
Merged sdkconfig from Common + fragments + SysType (and the build variant's files) |
build_variant.txt |
The build variant the sdkconfig was generated for |
sdkconfig |
Final ESP IDF sdkconfig (copy of merged defaults) |
SysTypes.json.merged |
Merged SysType JSON from Common + SysType |
SysTypeInfoRecs.h |
Generated C++ header with SysType JSON as string literals |
DeviceTypeRecords_generated.h |
Generated I2C device type records for DeviceManager |
DevicePollRecords_generated.h |
Generated poll result decode structures |
BuildWebUI/ |
Output of the WebUI npm build (gzipped HTML/JS/CSS) |
FSImage/ |
Assembled file system contents (FSImage + WebUI merged) |
RaftCore/ |
Fetched RaftCore source (if not using raftdevlibs) |
RaftSysMods/ |
Fetched RaftSysMods source (if not using raftdevlibs) |
| (other libraries) | Any other fetched Raft component sources |
All build scripts live in the scripts/ directory of RaftCore:
| Script | Language | Purpose |
|---|---|---|
RaftBootstrap.cmake |
CMake | SysType resolution, features.cmake, library fetch |
RaftBootstrapPhase2.cmake |
CMake | Config merging, header gen, ESP IDF integration, partitions, WebUI, FS image |
MergeSysTypeConfigs.py |
Python | Merges Common + SysType sdkconfig.defaults (with any sdkconfig fragments) and SysTypes.json files |
GenerateSysTypes.py |
Python | Converts merged SysTypes.json to SysTypeInfoRecs.h (C++ header with versioning) |
ProcessDevTypeJsonToC.py |
Python | Converts DevTypes.json to DeviceTypeRecords and DevicePollRecords headers |
DecodeGenerator.py |
Python | Generates C++ decode functions and output structs from device type definitions (called by ProcessDevTypeJsonToC.py) |
PseudocodeHandler.py |
Python | Transpiles pseudocode custom decode programs to C++ (called by DecodeGenerator.py) |
GenWebUI.py |
Python | Runs npm build on WebUI source, gzip-compresses output |
RaftGenFSImage.cmake |
CMake | Creates LittleFS or SPIFFS binary image from the assembled FSImage folder |
MyProject/
├── CMakeLists.txt ← Stage 2: downloads RaftBootstrap
├── Dockerfile ← Docker build image (optional, for raft build)
├── components/
│ └── MySysMod/ ← User application SysMod(s)
├── main/ ← ESP IDF main component
├── systypes/
│ ├── Common/
│ │ ├── SysTypes.json ← Base configuration JSON
│ │ ├── sdkconfig.defaults ← Base SDK config
│ │ ├── features.cmake ← Build settings (IDF_TARGET, RAFT_COMPONENTS, etc.)
│ │ ├── partitions.csv ← Flash partition table
│ │ ├── FSImage/ ← Static file system contents
│ │ ├── WebUI/ ← Web UI source (npm project)
│ │ └── DevTypes.json ← Device type definitions (optional)
│ └── MySysType/
│ ├── SysTypes.json ← SysType-specific overrides
│ ├── sdkconfig.defaults ← SysType-specific SDK config overrides
│ └── features.cmake ← Usually just includes Common/features.cmake
├── raftdevlibs/ ← Local library overrides (optional, gitignored)
│ └── RaftCore/ ← Local checkout for development
└── build/
├── raft.info ← RaftCLI build state
├── raft/
│ └── partitions.csv ← Copied partition table
└── MySysType/
├── raft/ ← Generated build artifacts
│ ├── sdkconfig.defaults.merged
│ ├── sdkconfig
│ ├── SysTypes.json.merged
│ ├── SysTypeInfoRecs.h
│ ├── DeviceTypeRecords_generated.h
│ ├── DevicePollRecords_generated.h
│ ├── BuildWebUI/
│ ├── FSImage/
│ ├── RaftCore/ ← Fetched library source
│ └── RaftSysMods/ ← Fetched library source
├── compile_commands.json
└── ... ← ESP IDF build output
Common Tasks
Getting Started
- Quick Start
- Architecture at a Glance
- Writing Your First SysMod
- Adding a Comms Channel
- Adding an I2C Device Type
Scaffolding & Building
- Raft CLI
- Web Apps (Device Types, Flasher)
- SysTypes
- Top-Level SysType
- Build Process
- WebUI Build Pipeline
- File System
- Partitions & Flash
- Local Dev Libraries
- Library Developer Guide
Architecture
Built-in SysMods
- NetworkManager
- WiFi Troubleshooting
- BLEManager
- WebServer
- MQTTManager
- SerialConsole
- CommandSerial
- CommandSocket
- CommandFile
- FileManager
- LogManager
- ESPOTAUpdate
- StatePublisher
- Remote Logging
- Data Source Registration
Comms & Protocols
- Stack Overview
- Comms Channels
- ProtocolExchange
- RICREST Protocol
- Real-Time Streams
- Adding REST Endpoints
- Built-in REST Endpoints
- File Download (OKTO)
- OTA Update Flow
Devices & Buses
- DeviceManager
- Device Manager REST API
- Device Factory & Classes
- Device Type Records
- Selecting Device Types
- Adding an I2C Device Type
- Device Data Publishing
- Data Logger
- I2C Bus
- I2C Device Scanning
- I2C ID & Polling
- MotorControl Overview
- MotorControl Config
- MotorControl Commands
- Camera
ROS 2
Helpers
Reference