Use mimalloc to prevent OOM in docker environments. - #139
Conversation
| } | ||
| // The grid z-extent comes from the geometry while indexed_bounds comes | ||
| // from the index; they agree to float-path precision. | ||
| assert!((world.grid.bbox[2] - indexed_bounds.min_z).abs() < 1e-6); |
There was a problem hiding this comment.
New f32 arithmetic causes a very small deviation, perhaps we should add a new default epsilon to validate against.
There was a problem hiding this comment.
Ok, yes, we could add a const with the epsilon, like DEFAULT_FLOAT_EPSILON, and use that
|
@rijos thanks for this, I didn't realise that memory usage would be this bad. You are using regular CityJSON files (not cityjsonseq etc) as input, right? |
| let filtered = feature_filter.apply(&model)?; | ||
| model = filtered.model; | ||
| // A package with no retained geometry yields `model: None`. | ||
| let Some(model) = filtered.model else { |
There was a problem hiding this comment.
I dislike this silent edge case, either log or abort.
We're using regular CityJSON files as input. |
2 participants
I noticed our tyler workers being killed by docker while processing the terrain dataset, by switching the allocator to mimalloc we prevent this behavior by releasing the pages earlier to the OS. Under normal circumstances the glibc allocator is fine.
Before
After
This is part one of a two-part fix, this fixes going OOM while generating the glb's. The second place where we probably go OOM is indexing all generated glb's. I'll keep you posted.
Created part two: 3DGI/cityjson-rs#15
Additional note, a lot of rework has been done in cityjson-rs to prevent tyler going OOM. the new streaming reindex works for terrain, so I've decided to incorporate the public cityjson-rs api changes in this pull request for tyler so we can make use of these changes.