Reducing Tor memory usage when running in Shadow

[robgjansen]

In case you missed it, an excellent series of blog posts was published back in January sharing a detailed analysis of Tor's memory usage and which methods work and don't work for reducing it.

• https://1aeo.com/blog/maxconsensusagefordiffs-limiting-for-guard-memory.html
• https://1aeo.com/blog/periodic-restarts-workaround-guard-memory.html
• https://1aeo.com/blog/maxmeminqueues-myth-for-guard-memory.html
• https://1aeo.com/blog/dircache-memory-trade-off-tor-guard-relays.html
• https://1aeo.com/blog/memory-allocators-tor-relay-fragmentation-with-custom-allocators.html
• https://1aeo.com/blog/mimalloc-209-tor-relay-deployment.html

The summary article gives an overview of their analysis:
https://1aeo.com/blog/tor-memory-optizations-what-actually-works.html

I think there are useful insights here that we might be able to use to reduce memory usage in Shadow.

Key Findings

• The Culprit: Memory fragmentation in glibc's allocator, caused by high churn of Directory Cache objects
• False Hopes: Standard config tweaks like MaxMemInQueues don't fix this. Disabling DirCache works but disqualifies you as a Guard
• The Fix: Changing the memory allocator is the only viable solution

Thus, two things worth exploring:

1. Most interesting for Shadow: "Disabling DirCache works but disqualifies you as a Guard". That's a problem in the live network. BUT, in Shadow, we hard code the guard flag vote, so I'm thinking relays should get the guard flag no matter what. So, does that mean setting DirCache 0 would be a viable fix in our Shadow experiments?

2. They found good results when changing the memory allocator. They found the best allocator was mimalloc 2.0.9. (Some of their graphs look suspiciously similar to our patterns of memory growth when running Tor in Shadow!) They replaced the allocator using LD_PRELOAD=/usr/local/lib/mimalloc/libmimalloc-2.0.9.so. I'm not quite sure how that would interact with Shadow's preloads - could we place mimalloc in the correct order in our preload list such that it could work correctly for us?

[sporksmith]

So, does that mean setting DirCache 0 would be a viable fix in our Shadow experiments?

Sounds plausible!

I'm not quite sure how that would interact with Shadow's preloads - could we place mimalloc in the correct order in our preload list such that it could work correctly for us?

Maybe!

If we just add the LD_PRELOAD in the shadow config file env vars, and shadow interposes it normally, it might just work, though we might run into issues like shadow/shadow#3763. Hopefully that would become moot once it's fixed (perhaps by shadow/shadow#3765)