Merge pull request #39 from PyNumLab/feature/compiler-preprocessing

Implement compiler preprocessing module for handling includes in Fort…

Author: saidctb

AGENTS.md

@@ -11,4 +11,6 @@ Ignore:
 - *.json
 
 Do not spend context window or analysis on those files unless explicitly requested.
+When updating tests, remove obsolete tests that only assert removed/old implementation behavior does not exist. Do not preserve rejection or absence checks for API/features that were intentionally removed unless explicitly requested.
+When investigating coverage failures, mirror the GitHub Actions workflow before deciding the fix: run coverage with `COVERAGE_PROCESS_START=pyproject.toml`, combine parallel data with `python -m coverage combine`, then run `python -m coverage report`. Do not assume a plain local coverage run matches CI, especially when subprocess tests are involved.
 When you create a commit add this prefix to the message to know that you did push the commit "codex: ..."

README.md

@@ -55,7 +55,8 @@ Current handled coverage:
 The C frontend is currently parse-only. It supports:
 
 - Raw-source directive metadata for includes, simple macros, conditionals, and
-  pragmas.
+  pragmas. Raw mode records these facts but does not expand macros or select
+  conditional branches.
 - Compiler-assisted preprocessing through the shared CLI flags, with `#line`
   and GCC/Clang linemarker remapping back to original source locations.
 - Top-level variables, typedefs, function declarations/definitions, structs,
@@ -66,8 +67,9 @@ The C frontend is currently parse-only. It supports:
 - Project include/index facts through `parse_c_project(...)`, with includes
   recorded non-recursively: only explicitly supplied files or files below an
   explicitly supplied directory are parsed.
-- Raw mutually exclusive function alternatives preserved for later semantic
-  selection rather than collapsed into one signature.
+- Compiler mode is the wrapper-facing path for macro-dependent APIs: it parses
+  one compiler-expanded translation unit and keeps mutually exclusive branches
+  separate across build configurations.
 
 The supported C subset continues through semantic IR conversion, `.pyi`
 generation, and wrap-readiness.
@@ -76,11 +78,16 @@ generation, and wrap-readiness.
 
 Public API entrypoints include:
 
-- `x2py.parse_fortran_file(source_or_path, filename=None, macro_defines=None, encoding="utf-8") -> FortranFile`
+- `x2py.parse_fortran_file(source_or_path, filename=None, encoding="utf-8") -> FortranFile`
 - `x2py.parse_fortran_project(files, encoding="utf-8") -> FortranProject`
-- `x2py.parse_c_file(source_or_path, filename=None, macro_defines=None, include_dirs=None, preprocessing="raw", encoding="utf-8") -> CFile`
-- `x2py.parse_c_project(files, include_dirs=None, macro_defines=None, preprocessing="raw", encoding="utf-8") -> CProject`
+- `x2py.parse_c_file(source_or_path, filename=None, include_dirs=None, preprocessing="raw", encoding="utf-8") -> CFile`
+- `x2py.parse_c_project(files, include_dirs=None, preprocessing="raw", encoding="utf-8") -> CProject`
 - `x2py.fortran_file_to_semantic_modules(parsed_file, standalone_module_name=None) -> list[SemanticModule]`
+- `x2py.fortran_project_to_semantic_modules(project) -> list[SemanticModule]`
+- `x2py.c_file_to_semantic_modules(parsed_file) -> list[SemanticModule]`
+- `x2py.c_project_to_semantic_modules(project) -> list[SemanticModule]`
+- `x2py.emit_module_stubs(module_or_modules) -> dict[str, str]`
+- `x2py.load_pyi_modules(path_or_paths, encoding="utf-8") -> list[SemanticModule]`
 - `x2py.assess_semantic_wrap_readiness(semantic_ir, source=None) -> dict`
 - `x2py.assess_pyi_wrap_readiness(path_or_paths, encoding="utf-8") -> dict`
 - `x2py.c_type_probe.probe_c_standard_types(config, runner=None) -> CStandardTypeProbeReport`
@@ -152,7 +159,15 @@ python -m x2py path/to/c_src --language c --parse
 Fortran directories scan `.f`, `.for`, `.ftn`, `.f90`, `.f95`, `.f03`,
 `.f08`; C directories scan `.c`, `.h`, and `.i` files.
 
-### Compiler preprocessing and target probes
+### Compiler preprocessing, includes, and target probes
+
+Wrapper-facing source parsing should use compiler preprocessing whenever the
+input contains C/CPP preprocessing. The selected compiler is authoritative for
+macro expansion, `#if`/`#ifdef` branch selection, C `#include`, Fortran CPP
+`#include`, predefined macros, `-D`/`-U`, include paths, target flags, and
+sysroot behavior. Internal parser mode remains available for plain source,
+already-preprocessed source, and focused parser tests; it does not evaluate CPP
+branches.
 
 The shared compiler mode is:
 
@@ -168,9 +183,45 @@ python -m x2py path/to/source.f90 --language fortran --parse \
 ```
 
 For C, `--language c --preprocess compiler` runs the exact compiler
-preprocessor and parses stdout. C also supports `--compile-commands
-build/compile_commands.json`; the matching entry supplies the compiler and
-project flags.
+preprocessor and parses stdout. C and Fortran can use `--compile-commands
+build/compile_commands.json` when a matching entry supplies the compiler and
+project flags. GCC-compatible C/Clang invocations use `-E -x c`; GNU Fortran
+invocations use `-E -cpp`. Linemarkers are preserved so parser locations can be
+mapped back to original files. For unsupported compiler families, use
+`--preprocessor-adapter command-template --preprocess-template '...'`; the
+minimum adapter contract is expanded source on stdout.
+
+Fortran native `include "file.inc"` is resolved after compiler CPP output and
+before parsing. This is textual insertion into the current module, procedure,
+interface, or execution scope; it is not the same as `use module_name`. Native
+includes are resolved relative to the including file first, then configured
+`-I` directories, and duplicate textual inclusion is preserved. Missing
+includes and cycles are reported as preprocessing diagnostics.
+
+Preprocessing JSON records the exact recipe: compiler or adapter, argv, working
+directory, include directories, defines, undefs, standard, extra compiler
+arguments, included files, source mappings, diagnostics, and optional macro
+metadata when the adapter output exposes it. System-header declarations are
+classified private by default. Reachable project includes are public by
+default; use `--include-exposure roots-only`, `--public-include`, and
+`--private-include` to control wrapper export. Private declarations remain
+available internally for type resolution. Public signatures that refer to
+private C handle types can use private opaque classes rather than exposing data
+members.
+
+The C parser tolerates common compiler-expanded declaration syntax from system
+headers, including GNU attributes, `__declspec(...)`, alternate qualifier
+spellings, declaration-level `asm(...)`, calling-convention keywords,
+`typeof(...)`, `_BitInt(...)`, and selected extended scalar names. Harmless
+syntax is accepted without exposing private header declarations. Ignored
+extensions that can affect ABI, layout, symbol identity, or type identity
+produce `C_UNMODELED_COMPILER_EXTENSION` warnings.
+
+Preprocessing failures print explicit categories such as
+`PREPROCESSOR_NOT_FOUND`, `PREPROCESSOR_FAILED`,
+`INVALID_COMPILER_ARGUMENTS`, `UNSUPPORTED_COMPILER_CAPABILITY`,
+`PROVENANCE_UNAVAILABLE`, `INCLUDE_NOT_FOUND`, and `INCLUDE_CYCLE` without a
+Python traceback. Pass `--debug` to re-raise and show the traceback.
 
 Target-dependent type facts are not hard-coded. They are probed with the same
 compiler path and target-relevant flags because results may change with ABI,
@@ -757,3 +808,46 @@ source/target mapping. A non-renamed `use iso_c_binding, only: c_int` maps
 `source="delete_input_list"` and `target="delete_input"`. The semantic layer
 uses that information to emit Python stub imports such as
 `from list_input import delete_input_list as delete_input`.
+
+Fortran `use` dependencies are not parsed or wrapped recursively. If a
+procedure refers to an imported derived type, semantic IR records its defining
+module and represents the reference as an opaque handle unless the defining
+module is explicitly part of the wrapping target. Explicitly supplied modules
+share one wrapped-type registry, so the imported reference resolves to the
+single class emitted by its owner module without being re-exported by the
+importing module. Reachable include exposure is already handled separately by
+the preprocessing include policy; a future dependency-expansion option would
+apply specifically to recursive Fortran `use` traversal.
+
+When an imported derived type remains external, `.pyi` generation emits an
+owner-module dependency stub. For example, wrapping only `physics.f90` may
+produce:
+
+```python
+# physics.pyi
+from types_mod import particle
+
+def move(p: Ptr(particle)) -> None: ...
+```
+
+```python
+# types_mod.pyi
+class particle(Opaque):
+    pass
+```
+
+`python -m x2py physics.f90 --pyi --out` writes both files beside the source.
+`load_pyi_modules(...)` loads a file set or directory, preserves opaque classes,
+and reconciles imported references against edited owner stubs. Replacing the
+opaque placeholder with a concrete edited class changes the semantic reference
+from `representation="opaque"` to `representation="wrapped"`. Existing
+`Annotated[...]` constraints also round-trip through this editable interface;
+richer coercion syntax can be added to the same `.pyi` format later.
+
+The same opaque-handle file-set model applies to C. A local forward declaration
+such as `struct context;` emits `class context(Opaque): pass`. When a public C
+header uses a struct from another explicitly supplied header, its generated
+stub imports the class from that header's stub. A private included struct used
+through a public pointer boundary emits an opaque owner-module dependency stub.
+An unresolved C typedef is left unresolved rather than guessed to be opaque,
+because its ABI may not be pointer-shaped.

c_parser/cli.py

@@ -9,7 +9,7 @@
 from pathlib import Path
 from typing import Any
 
-from .models import CFile, CParseError, c_model_to_dict
+from .models import CFile, CMacro, CParseError, CSourceLocation, c_model_to_dict
 from .parser import CParser
 
 
@@ -44,6 +44,38 @@ def expand_c_paths(paths: list[str]) -> list[Path]:
     return sorted(set(expanded))
 
 
+def attach_preprocessing_recipe(parsed: CFile, preprocessing_recipe: dict[str, Any] | None) -> None:
+    """Attach compiler recipe side-channel facts to a parsed C file."""
+
+    parsed.preprocessing_recipe = preprocessing_recipe
+    if not preprocessing_recipe:
+        return
+    existing = {(macro.name, macro.source_location.filename if macro.source_location else None, macro.source_location.line if macro.source_location else None) for macro in parsed.macros}
+    for item in preprocessing_recipe.get("macros") or []:
+        if not isinstance(item, dict):
+            continue
+        name = item.get("name")
+        if not isinstance(name, str) or not name:
+            continue
+        location = CSourceLocation(
+            filename=item.get("path") if isinstance(item.get("path"), str) else None,
+            line=item.get("line") if isinstance(item.get("line"), int) else None,
+            column=1,
+        )
+        key = (name, location.filename, location.line)
+        if key in existing:
+            continue
+        parsed.macros.append(
+            CMacro(
+                name=name,
+                value=item.get("value") if isinstance(item.get("value"), str) else None,
+                function_like=bool(item.get("function_like")),
+                source_location=location,
+            )
+        )
+        existing.add(key)
+
+
 def parse_c_report(
     paths: list[str],
     *,
@@ -70,7 +102,7 @@ def parse_c_report(
                 include_dirs=include_dirs,
                 preprocessing=preprocessing,
             )
-            parsed.preprocessing_recipe = preprocessing_recipe
+            attach_preprocessing_recipe(parsed, preprocessing_recipe)
         out[str(p)] = c_model_to_dict(parsed)
     return out
 

c_parser/lexer.py

@@ -93,6 +93,7 @@ class CTopLevelSegment:
 _LINE_DIRECTIVE_RE = re.compile(
     r'^\s*#\s*line\s+(?P<line>\d+)(?:\s+(?:"(?P<quoted>(?:[^"\\]|\\.)*)"|(?P<bare>\S+)))?'
 )
+_AGGREGATE_HEADER_ATTRIBUTE_RE = re.compile(r"\b(?:__attribute__?|__declspec(?:__)?)\b")
 
 
 def _source_line(lines: list[str], line_number: int) -> str | None:
@@ -317,18 +318,76 @@ def top_level_partition(text: str, delimiter: str = "=") -> tuple[str, str | Non
     return text.strip(), None
 
 
-def _is_aggregate_definition_header(header: str) -> bool:
+def _balanced_invocation_end(text: str, open_index: int) -> int | None:
+    """Return the offset after one balanced parenthesized invocation."""
+    depth = 0
+    quote = ""
+    escaped = False
+    for index in range(open_index, len(text)):
+        char = text[index]
+        if quote:
+            if escaped:
+                escaped = False
+            elif char == "\\":
+                escaped = True
+            elif char == quote:
+                quote = ""
+            continue
+        if char in {'"', "'"}:
+            quote = char
+        elif char == "(":
+            depth += 1
+        elif char == ")":
+            depth -= 1
+            if depth == 0:
+                return index + 1
+    return None
+
+
+def _strip_aggregate_header_attributes(header: str) -> str:
+    """Blank attributes that can appear between an aggregate keyword and tag."""
+    characters = list(header)
+    for match in _AGGREGATE_HEADER_ATTRIBUTE_RE.finditer(header):
+        end = match.end()
+        open_index = end
+        while open_index < len(header) and header[open_index].isspace():
+            open_index += 1
+        if open_index < len(header) and header[open_index] == "(":
+            end = _balanced_invocation_end(header, open_index) or end
+        for index in range(match.start(), end):
+            if characters[index] != "\n":
+                characters[index] = " "
+    return "".join(characters)
+
+
+def _is_aggregate_definition_header(
+    header: str,
+    *,
+    tolerate_compiler_extensions: bool = False,
+) -> bool:
     """Identify a tag definition before deciding that a brace starts a body."""
+    if tolerate_compiler_extensions:
+        header = _strip_aggregate_header_attributes(header)
     compact = " ".join(header.split())
     if "(" in compact or "=" in compact:
         return False
     words = compact.split()
     return any(word in {"struct", "union", "enum"} for word in words)
 
 
-def _is_braced_declaration_header(header: str) -> bool:
+def _is_braced_declaration_header(
+    header: str,
+    *,
+    tolerate_compiler_extensions: bool = False,
+) -> bool:
     """Return whether a brace belongs to a declaration preserved through `;`."""
-    return _is_aggregate_definition_header(header) or top_level_partition(header, "=")[1] is not None
+    return (
+        _is_aggregate_definition_header(
+            header,
+            tolerate_compiler_extensions=tolerate_compiler_extensions,
+        )
+        or top_level_partition(header, "=")[1] is not None
+    )
 
 
 def split_top_level_c_source(
@@ -337,6 +396,7 @@ def split_top_level_c_source(
     *,
     skip_preprocessor: bool = True,
     use_linemarkers: bool = False,
+    tolerate_compiler_extensions: bool = False,
 ) -> list[CTopLevelSegment]:
     """Split C source into top-level declarations and definition headers."""
     stripped = strip_c_comments(source)
@@ -420,7 +480,10 @@ def split_top_level_c_source(
                     block_start_line = start_line
                     block_start_column = start_column
                     block_source_line = start_mapping.source_line
-                    braced_declaration = _is_braced_declaration_header(header)
+                    braced_declaration = _is_braced_declaration_header(
+                        header,
+                        tolerate_compiler_extensions=tolerate_compiler_extensions,
+                    )
             brace_depth = 1
             if not braced_declaration:
                 start_index = None