`std::text_encoding`

Executive Abstract

std::text_encoding is a class template introduced in C++26 in header <text_encoding> (standardized via proposal P1885R12) that provides a standard, platform-agnostic type for identifying, querying, and comparing character encodings. It bridges C++ applications with the official IANA (Internet Assigned Numbers Authority) Character Sets registry.

Before C++26, C++ programs lacked a standard way to determine the character encoding used for C++ string literals at compile time (std::text_encoding::literal()) or the system/terminal execution environment at runtime (std::text_encoding::environment()). std::text_encoding solves this by encapsulating IANA MIB (Management Information Base) enum identifiers, standard canonical names, and alias views (.aliases()) while providing normalized string comparison logic that accounts for historical character encoding name variations across operating systems.

Motivation & Historical Context

Pre-C++26 Deficiencies: The Locale & Encoding Ambiguity

Historically, standard C++ tied character set assumptions directly to std::locale. However, modern internationalization practice separates encodings (mappings between bit patterns/code units and Unicode code points) from locales (formatting rules for dates, currencies, and collation orders).

In multi-platform or network-aware applications, developer assumptions regarding string encodings frequently caused quiet data corruption:

// Pre-C++26 Ambiguity: How is "Hello \u00E9" encoded in memory?
const char* text = "Hello \u00E9";

// FLAW 1: Compilers disagreed on character literal encodings!
// GCC/Clang on Linux defaults to UTF-8.
// MSVC on Windows traditionally defaulted to local ANSI code pages (e.g., Windows-1252 or CP932).

// FLAW 2: No standard compile-time query!
// A library couldn't write static_assert checks ensuring string literals were UTF-8 encoded.

// FLAW 3: OS-specific encoding name variations!
// Linux uses "UTF-8", macOS uses "UTF-8", Windows uses "CP_UTF8" / "CP65001",
// and iconv might expect "UTF8" or "csUTF8".

Deficiencies of Pre-C++26 Text Encodings:

Locale Monolith Overload: Querying std::locale("").name() returned platform-dependent locale strings (e.g., "en_US.UTF-8" on Posix vs "en-US" on Windows) that failed to reliably reveal underlying byte encodings.
Compile-Time Opaque Literals: Code templates and constexpr algorithms could not verify whether ordinary character literals ("...") matched expected character sets like UTF-8 or ASCII.
Inconsistent OS Encoding Names: System APIs referred to identical encodings using mismatched string names, forcing cross-platform code bases to maintain fragile custom lookup tables for I/O and text conversion libraries (iconv, ICU, Win32 MultiByteToWideChar).

Solution: ISO C++26 introduced std::text_encoding (P1885R12), mapping encodings directly to standardized IANA MIB integer codes and providing normalized lookup functions.

Technical Mechanics & Architecture

1. Structure of `std::text_encoding`

                                  std::text_encoding Object
   ┌──────────────────────────────────────────────────────────────────────────────────┐
   │ 1. MIB Enum Identifier (std::text_encoding::id)                                  │
   │    • E.g., id::UTF8 (106), id::ASCII (3), id::ShiftJIS (17)                      │
   ├──────────────────────────────────────────────────────────────────────────────────┤
   │ 2. Canonical IANA Name (const char*)                                             │
   │    • E.g., "UTF-8", "US-ASCII", "Shift_JIS"                                      │
   ├──────────────────────────────────────────────────────────────────────────────────┤
   │ 3. Aliases Range View (text_encoding::aliases_view)                              │
   │    • E.g., {"utf-8", "utf8", "csUTF8", "unicode-1-1-utf-8"}                      │
   └──────────────────────────────────────────────────────────────────────────────────┘

2. Standard MIB Enum Identifiers (`std::text_encoding::id`)

std::text_encoding::id is an enum class containing standard IANA character set enumerators:

namespace std {
    struct text_encoding {
        enum class id : std::int_least32_t {
            other = 1,
            unknown = 2,
            ASCII = 3,
            ISOLatin1 = 4,
            ShiftJIS = 17,
            UTF8 = 106,
            UTF16BE = 101,
            UTF16LE = 102,
            UTF32BE = 104,
            UTF32LE = 105,
            // ... complete IANA registry mapping
        };
        using enum id;
    };
}

3. Case-Insensitive Normalized Comparison Rules (`comp-name`)

Comparing two std::text_encoding objects or matching string names against encodings follows the standard normalized IANA rules:

Strips all non-alphanumeric characters (hyphens, underscores, spaces).
Converts all alphabetic characters to lowercase.
Strips standalone leading zeros that do not immediately follow a numeric prefix.

This ensures that "UTF-8", "utf8", "u.t.f.--8", and "csUTF8" evaluate as equal under std::text_encoding comparison operators.

Evolution Across Standards (C++11 to C++26/C++29)

┌────────────────────────────────────────────────────────────────────────┐
│ C++11 / C++14 / C++17 / C++20 / C++23                                  │
├────────────────────────────────────────────────────────────────────────┤
│ • Encodings were implicitly tied to std::locale.                       │
│ • No standard compile-time reflection for literal character encodings. │
│ • Cross-platform text libs relied on OS-specific code-page queries.    │
└────────────────────────────────────────────────────────────────────────┘
                                   │
                                   ▼
┌────────────────────────────────────────────────────────────────────────┐
│ C++26                                                                  │
├────────────────────────────────────────────────────────────────────────┤
│ • Introduced <text_encoding> header and std::text_encoding class.      │
│ • Added static consteval std::text_encoding::literal() for compile-    │
│   time literal validation.                                             │
│ • Added static std::text_encoding::environment() for runtime execution │
│   encoding queries.                                                    │
│ • Added std::locale::encoding() member function integration.           │
└────────────────────────────────────────────────────────────────────────┘

Comprehensive Code Examples

Example 1: Compile-Time Literal Validation & Environment Query

Demonstrates using std::text_encoding::literal() at compile time with static_assert to enforce UTF-8 build settings and querying runtime environment encodings.

#include <iostream>
#include <locale>

// Feature check macro for C++26 <text_encoding>
#if defined(__cpp_lib_text_encoding) && __cpp_lib_text_encoding >= 202306L
#include <text_encoding>

void print_encoding_info(const char* label, const std::text_encoding& enc) {
    std::cout << label << ": " << enc.name()
              << " (IANA MIB Code: " << static_cast<int>(enc.mib()) << ")\n";
    std::cout << "  Known Aliases:\n";
    for (const char* alias : enc.aliases()) {
        std::cout << "   - " << alias << "\n";
    }
}

int main() {
    // 1. Compile-Time Query: Literal Encoding
    constexpr std::text_encoding literal_enc = std::text_encoding::literal();

    // Verify at compile-time that string literals are using UTF-8
    static_assert(literal_enc.mib() == std::text_encoding::id::UTF8 ||
                  literal_enc.mib() != std::text_encoding::id::unknown,
                  "Compilation requires known string literal encoding!");

    print_encoding_info("Compile-Time Literal Encoding", literal_enc);

    std::cout << "\n----------------------------------------\n";

    // 2. Runtime Query: OS Execution Environment Encoding
    std::text_encoding env_enc = std::text_encoding::environment();
    print_encoding_info("Runtime Environment Encoding", env_enc);

    // 3. Runtime Query: C++ Locale Encoding
    std::text_encoding loc_enc = std::locale("").encoding();
    print_encoding_info("Default Locale Encoding", loc_enc);

    // 4. Case-insensitive equality check
    if (env_enc == std::text_encoding::id::UTF8) {
        std::cout << "\nSUCCESS: Runtime environment fully supports UTF-8!\n";
    }

    return 0;
}
#else
int main() {
    std::cout << "std::text_encoding is demonstrated for C++26 standard environments.\n";
    return 0;
}
#endif

Example 2: Integration with Conversion Pipeline (e.g., `iconv` / System Transcoding)

Demonstrates using std::text_encoding to dynamically select and validate encoding parameters for text processing pipelines.

#include <iostream>
#include <string_view>

#if defined(__cpp_lib_text_encoding) && __cpp_lib_text_encoding >= 202306L
#include <text_encoding>

bool is_utf8_compatible(std::string_view user_encoding_name) {
    // Construct std::text_encoding from string name
    std::text_encoding requested{user_encoding_name};

    // Check if MIB matches UTF-8 directly using normalized comparison
    return requested == std::text_encoding::id::UTF8;
}

int main() {
    // Test multiple equivalent encoding names used across OS platforms
    constexpr std::array<const char*, 4> test_names{
        "UTF-8", "utf8", "csUTF8", "Windows-1252"
    };

    for (const char* name : test_names) {
        bool match = is_utf8_compatible(name);
        std::cout << "Encoding name '" << name << "' is UTF-8 compatible: "
                  << std::boolalpha << match << "\n";
    }

    return 0;
}
#else
int main() { return 0; }
#endif

Engineering Best Practices & Common Pitfalls

Use std::text_encoding::literal() in static_assert Guards:
- If your codebase assumes character literals ("...") are UTF-8, write a static_assert(std::text_encoding::literal() == std::text_encoding::id::UTF8) at the application entry point to catch improper compiler flags (e.g., missing -fexec-charset=UTF-8 on GCC or /execution-charset:utf-8 on MSVC).
Decouple Encodings from Locales:
- Avoid using std::locale solely to discover text encoding. Use std::text_encoding::environment() for system I/O streams and std::text_encoding::literal() for string literals.
Rely on operator== for Encoding Matching:
- Do not perform custom std::string comparisons against encoding names. std::text_encoding::operator== automatically applies the standard IANA normalized comparison rules (comp-name), making string comparison robust across capitalization and special characters.
Handle id::unknown and id::other Gracefully:
- id::unknown means the system could not determine the encoding. id::other means the encoding is known by string name but lacks a registered IANA MIB integer code. Always provide fallback code paths when querying dynamic user environments.

Relationships

Related to:
- std::format / std::print — Formatting and printing Unicode text safely to terminal outputs.
- std::string_view — Inspecting non-owning sequences of encoded code units.
- Unordered Containers — Standard hashing support (std::hash<std::text_encoding>) enables std::unordered_map storage.
Superseded by: None.
Supersedes: Platform-dependent nl_langinfo(CODESET), GetACP(), and fragile regex parsing of locale strings.

Back to C++26 section.

Navigation Menu

Repository navigation

stdtext_encoding.md

stdtext_encoding.md

`std::text_encoding`

Executive Abstract

Motivation & Historical Context

Pre-C++26 Deficiencies: The Locale & Encoding Ambiguity

Deficiencies of Pre-C++26 Text Encodings:

Technical Mechanics & Architecture

1. Structure of `std::text_encoding`

2. Standard MIB Enum Identifiers (`std::text_encoding::id`)

3. Case-Insensitive Normalized Comparison Rules (`comp-name`)

Evolution Across Standards (C++11 to C++26/C++29)

Comprehensive Code Examples

Example 1: Compile-Time Literal Validation & Environment Query

Example 2: Integration with Conversion Pipeline (e.g., `iconv` / System Transcoding)

Engineering Best Practices & Common Pitfalls

Relationships

FilesExpand file tree

stdtext_encoding.md

Latest commit

History

stdtext_encoding.md

File metadata and controls

std::text_encoding

Executive Abstract

Motivation & Historical Context

Pre-C++26 Deficiencies: The Locale & Encoding Ambiguity

Deficiencies of Pre-C++26 Text Encodings:

Technical Mechanics & Architecture

1. Structure of std::text_encoding

2. Standard MIB Enum Identifiers (std::text_encoding::id)

3. Case-Insensitive Normalized Comparison Rules (comp-name)

Evolution Across Standards (C++11 to C++26/C++29)

Comprehensive Code Examples

Example 1: Compile-Time Literal Validation & Environment Query

Example 2: Integration with Conversion Pipeline (e.g., iconv / System Transcoding)

Engineering Best Practices & Common Pitfalls

Relationships

`std::text_encoding`

1. Structure of `std::text_encoding`

2. Standard MIB Enum Identifiers (`std::text_encoding::id`)

3. Case-Insensitive Normalized Comparison Rules (`comp-name`)

Example 2: Integration with Conversion Pipeline (e.g., `iconv` / System Transcoding)