Skip to content

Commit 44934ce

Browse files
authored
Add dynamic model generation support (#2901)
* Add dynamic model generation support * Add dynamic model generation support * Move dynamic model generation to separate module with lazy import * Move imports to top level except circular import avoidance * Add create_module_getattr helper for DRY lazy imports * Move dynamic.py imports to top level and use direct import in __init__.py * Add multi-module support and simplify thread lock * Refactor tests with assert helpers and DRY patterns * Refactor to use stable_toposort and ast-based import parsing * Refactor tests to use model_validate assertions * Add generate_and_validate helper with inline-snapshot * Use assert_dynamic_models helper with external_file * Fix test helper to use external_file for JSON comparison * Remove inline comment from test helper * Remove inline comments and refactor with walrus operator * Extract OpenAPI schema to external file * Add tests to achieve 100% coverage for dynamic.py * Fix thread safety issues in dynamic model generation * Refactor cache key to preserve list order and remove dead code * Make dynamic module imports lazy * Fix partial coverage in test cleanup * Add documentation for dynamic model generation * Add dynamic model generation to docs navigation
1 parent a2b2756 commit 44934ce

22 files changed

Lines changed: 1215 additions & 11 deletions

docs/dynamic-model-generation.md

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
# Dynamic Model Generation
2+
3+
Generate real Python model classes from JSON Schema or OpenAPI at runtime without writing files.
4+
5+
## Overview
6+
7+
While `generate()` produces source code as strings, `generate_dynamic_models()` creates actual Python classes that you can use immediately for validation and data processing. This is useful for:
8+
9+
- Runtime schema validation without code generation step
10+
- Dynamic API clients that adapt to schema changes
11+
- Testing and prototyping
12+
- Plugin systems with dynamic schemas
13+
14+
## Quick Start
15+
16+
```python
17+
from datamodel_code_generator import generate_dynamic_models
18+
19+
schema = {
20+
"type": "object",
21+
"properties": {
22+
"name": {"type": "string"},
23+
"age": {"type": "integer"}
24+
},
25+
"required": ["name"]
26+
}
27+
28+
models = generate_dynamic_models(schema)
29+
User = models["Model"]
30+
31+
# Use the model for validation
32+
user = User(name="Alice", age=30)
33+
print(user.model_dump()) # {'name': 'Alice', 'age': 30}
34+
35+
# Validation errors are raised
36+
try:
37+
User(age="not a number") # Missing required 'name', wrong type for 'age'
38+
except Exception as e:
39+
print(e)
40+
```
41+
42+
## API Reference
43+
44+
### `generate_dynamic_models()`
45+
46+
```python
47+
def generate_dynamic_models(
48+
input_: Mapping[str, Any],
49+
*,
50+
config: GenerateConfig | None = None,
51+
cache_size: int = 128,
52+
) -> dict[str, type]:
53+
```
54+
55+
**Parameters:**
56+
57+
| Parameter | Type | Default | Description |
58+
|-----------|------|---------|-------------|
59+
| `input_` | `Mapping[str, Any]` | required | JSON Schema or OpenAPI schema as dict |
60+
| `config` | `GenerateConfig \| None` | `None` | Generation options (same as `generate()`) |
61+
| `cache_size` | `int` | `128` | Maximum cached schemas. Set to `0` to disable |
62+
63+
**Returns:** `dict[str, type]` - Dictionary mapping class names to model classes.
64+
65+
### `clear_dynamic_models_cache()`
66+
67+
```python
68+
def clear_dynamic_models_cache() -> int:
69+
```
70+
71+
Clears the internal cache and returns the number of entries cleared.
72+
73+
## Examples
74+
75+
### JSON Schema with Nested Models
76+
77+
```python
78+
from datamodel_code_generator import generate_dynamic_models
79+
80+
schema = {
81+
"$defs": {
82+
"Address": {
83+
"type": "object",
84+
"properties": {
85+
"street": {"type": "string"},
86+
"city": {"type": "string"}
87+
},
88+
"required": ["street", "city"]
89+
},
90+
"Person": {
91+
"type": "object",
92+
"properties": {
93+
"name": {"type": "string"},
94+
"address": {"$ref": "#/$defs/Address"}
95+
},
96+
"required": ["name"]
97+
}
98+
},
99+
"$ref": "#/$defs/Person"
100+
}
101+
102+
models = generate_dynamic_models(schema)
103+
104+
# Both models are available
105+
Person = models["Person"]
106+
Address = models["Address"]
107+
108+
person = Person(
109+
name="Bob",
110+
address={"street": "123 Main St", "city": "NYC"}
111+
)
112+
print(person.model_dump())
113+
# {'name': 'Bob', 'address': {'street': '123 Main St', 'city': 'NYC'}}
114+
```
115+
116+
### OpenAPI Schema
117+
118+
OpenAPI schemas are auto-detected:
119+
120+
```python
121+
from datamodel_code_generator import generate_dynamic_models
122+
123+
openapi_schema = {
124+
"openapi": "3.0.0",
125+
"info": {"title": "User API", "version": "1.0.0"},
126+
"paths": {},
127+
"components": {
128+
"schemas": {
129+
"User": {
130+
"type": "object",
131+
"properties": {
132+
"id": {"type": "integer"},
133+
"email": {"type": "string", "format": "email"}
134+
},
135+
"required": ["id", "email"]
136+
}
137+
}
138+
}
139+
}
140+
141+
models = generate_dynamic_models(openapi_schema)
142+
User = models["User"]
143+
144+
user = User(id=1, email="alice@example.com")
145+
```
146+
147+
### With Custom Configuration
148+
149+
```python
150+
from datamodel_code_generator import generate_dynamic_models, GenerateConfig, DataModelType
151+
152+
schema = {"type": "object", "properties": {"name": {"type": "string"}}}
153+
154+
config = GenerateConfig(
155+
class_name="Customer",
156+
output_model_type=DataModelType.PydanticV2BaseModel,
157+
)
158+
159+
models = generate_dynamic_models(schema, config=config)
160+
Customer = models["Customer"]
161+
```
162+
163+
### Enum Models
164+
165+
```python
166+
from datamodel_code_generator import generate_dynamic_models
167+
168+
schema = {
169+
"type": "object",
170+
"properties": {
171+
"status": {
172+
"type": "string",
173+
"enum": ["pending", "approved", "rejected"]
174+
}
175+
}
176+
}
177+
178+
models = generate_dynamic_models(schema)
179+
Model = models["Model"]
180+
Status = models["Status"]
181+
182+
# Enum validation
183+
item = Model(status="approved")
184+
print(item.status) # Status.approved
185+
print(item.status.value) # 'approved'
186+
187+
# Invalid enum value raises error
188+
try:
189+
Model(status="invalid")
190+
except Exception as e:
191+
print(e)
192+
```
193+
194+
### Circular References
195+
196+
```python
197+
from datamodel_code_generator import generate_dynamic_models
198+
199+
schema = {
200+
"$defs": {
201+
"Node": {
202+
"type": "object",
203+
"properties": {
204+
"value": {"type": "string"},
205+
"children": {
206+
"type": "array",
207+
"items": {"$ref": "#/$defs/Node"}
208+
}
209+
}
210+
}
211+
},
212+
"$ref": "#/$defs/Node"
213+
}
214+
215+
models = generate_dynamic_models(schema)
216+
Node = models["Node"]
217+
218+
tree = Node(
219+
value="root",
220+
children=[
221+
Node(value="child1", children=[]),
222+
Node(value="child2", children=[
223+
Node(value="grandchild", children=[])
224+
])
225+
]
226+
)
227+
```
228+
229+
## Caching
230+
231+
Models are cached by schema content and configuration to avoid regeneration:
232+
233+
```python
234+
from datamodel_code_generator import generate_dynamic_models, clear_dynamic_models_cache
235+
236+
schema = {"type": "object", "properties": {"x": {"type": "integer"}}}
237+
238+
# First call generates models
239+
models1 = generate_dynamic_models(schema)
240+
241+
# Second call returns cached models (same object)
242+
models2 = generate_dynamic_models(schema)
243+
assert models1 is models2 # True
244+
245+
# Disable caching for specific call
246+
models3 = generate_dynamic_models(schema, cache_size=0)
247+
assert models1 is not models3 # True
248+
249+
# Clear all cached models
250+
cleared = clear_dynamic_models_cache()
251+
print(f"Cleared {cleared} cached schemas")
252+
```
253+
254+
## Thread Safety
255+
256+
`generate_dynamic_models()` is thread-safe. Multiple threads can safely call it concurrently:
257+
258+
```python
259+
import threading
260+
from datamodel_code_generator import generate_dynamic_models
261+
262+
schema = {"type": "object", "properties": {"x": {"type": "integer"}}}
263+
results = []
264+
265+
def worker():
266+
models = generate_dynamic_models(schema)
267+
results.append(models)
268+
269+
threads = [threading.Thread(target=worker) for _ in range(10)]
270+
for t in threads:
271+
t.start()
272+
for t in threads:
273+
t.join()
274+
275+
# All threads get the same cached models
276+
assert all(r is results[0] for r in results)
277+
```
278+
279+
## Limitations
280+
281+
| Limitation | Details |
282+
|------------|---------|
283+
| Pydantic v2 only | Pydantic v1 is not supported |
284+
| Not pickle-able | Use `model_dump()` to serialize instances |
285+
| Dict input only | Schema must be a `dict`, not a file path or string |
286+
287+
## Comparison with `generate()`
288+
289+
| Feature | `generate()` | `generate_dynamic_models()` |
290+
|---------|-------------|----------------------------|
291+
| Output | Source code string | Actual Python classes |
292+
| Use case | Code generation, file output | Runtime validation |
293+
| Caching | No | Yes (configurable) |
294+
| Thread-safe | Yes | Yes |
295+
| Pydantic v1 | Yes | No |
296+
297+
## See Also
298+
299+
- [Using as Module](using_as_module.md) - `generate()` function reference
300+
- [JSON Schema](jsonschema.md) - JSON Schema examples
301+
- [OpenAPI](openapi.md) - OpenAPI examples

docs/using_as_module.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -251,5 +251,6 @@ class Model(BaseModel):
251251

252252
## 📖 See Also
253253

254-
- 🖥️ [CLI Reference](cli-reference/index.md) - Complete CLI options (same parameters as module)
255-
- 📋 [Generate from JSON Schema](jsonschema.md) - JSON Schema examples
254+
- [Dynamic Model Generation](dynamic-model-generation.md) - Generate Python classes at runtime
255+
- [CLI Reference](cli-reference/index.md) - Complete CLI options (same parameters as module)
256+
- [Generate from JSON Schema](jsonschema.md) - JSON Schema examples

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,7 @@ lint.per-file-ignores."scripts/*.py" = [
184184
lint.per-file-ignores."tests/**/*.py" = [
185185
"FBT", # don't care about booleans as positional arguments in tests
186186
"INP001", # no implicit namespace
187+
"N806", # variable names for model classes can be uppercase
187188
"PLC0415", # local imports in tests are fine
188189
"PLC2701", # private import is fine
189190
"PLR0904", # too many public methods in test classes is fine

src/datamodel_code_generator/__init__.py

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -258,7 +258,7 @@ def chdir(path: Path | None) -> Iterator[None]:
258258
os.chdir(prev_cwd)
259259

260260

261-
def is_openapi(data: dict) -> bool:
261+
def is_openapi(data: Mapping[str, Any]) -> bool:
262262
"""Check if the data dict is an OpenAPI specification."""
263263
return "openapi" in data
264264

@@ -927,6 +927,23 @@ def infer_input_type(text: str) -> InputFileType:
927927
"`--input-file-type` option."
928928
)
929929

930+
931+
_LAZY_IMPORTS = {
932+
"clear_dynamic_models_cache": "datamodel_code_generator.dynamic",
933+
"generate_dynamic_models": "datamodel_code_generator.dynamic",
934+
}
935+
936+
937+
def __getattr__(name: str) -> Any:
938+
if name in _LAZY_IMPORTS:
939+
import importlib # noqa: PLC0415
940+
941+
module = importlib.import_module(_LAZY_IMPORTS[name])
942+
return getattr(module, name)
943+
msg = f"module {__name__!r} has no attribute {name!r}"
944+
raise AttributeError(msg)
945+
946+
930947
__all__ = [
931948
"DEFAULT_FORMATTERS",
932949
"DEFAULT_SHARED_MODULE_NAME",
@@ -959,5 +976,7 @@ def infer_input_type(text: str) -> InputFileType:
959976
"ReuseScope",
960977
"SchemaParseError",
961978
"TargetPydanticVersion",
979+
"clear_dynamic_models_cache", # noqa: F822
962980
"generate",
981+
"generate_dynamic_models", # noqa: F822
963982
]

0 commit comments

Comments
 (0)