fix: bound remote pubsub subscriptions to prevent memory exhaustion

[acul71]

Summary

• Fix memory exhaustion / DoS via unbounded remote topic subscriptions (GHSA-4f8r-922h-2vgv).
• Delete empty peer_topics entries on unsubscribe and peer removal (disconnect / blacklist).
• Enforce inbound RPC size (1 MiB) and per-RPC / per-peer subscription limits (5000 / 100 by default).

Fixes #1349

Related

• GHSA-4f8r-922h-2vgv
• Fix memory leak due to retention of empty topic maps go-libp2p-pubsub#710
• Security: GossipSub Memory Exhaustion via Unbounded Remote Topic Subscriptions cpp-libp2p#349

Test plan

• pytest tests/core/pubsub/test_pubsub_subscription_limits.py -v
• pytest tests/core/pubsub/test_pubsub.py::test_handle_subscription -v
• pytest tests/core/pubsub/test_pubsub.py::test_peers_subscribe -v
• make lint && make typecheck && make test && make linux-docs

Made with Cursor

[acul71]

Status update — GHSA-4f8r-922h-2vgv / #1349

cc @seetadev @sumanjeet0012 for review — this addresses the GossipSub subscription-flood memory exhaustion pattern reported across the libp2p ecosystem (js GHSA-4f8r-922h-2vgv, go go-libp2p-pubsub#710, cpp cpp-libp2p#349).

What’s been accomplished

Core mitigations (shared Pubsub layer — applies to GossipSub and FloodSub):

1. Empty topic map cleanup — peer_topics entries are removed when the last peer unsubscribes or disconnects (_remove_peer_from_topic, _clear_peer_from_all_topics), matching go-libp2p-pubsub Implement validation throttler for message validation in Pubsub #710.
2. Inbound RPC size limit — 1 MiB default on pubsub read (read_varint_prefixed_bytes + max_inbound_rpc_size), aligned with outbound DefaultMaxMessageSize.
3. Per-RPC subscription cap — default 5000 entries/RPC (js-libp2p post-15.0.23 default); over-limit RPCs are rejected.
4. Per-peer subscription cap — default 100 distinct topics/peer (go downstream / Filecoin-style limit); excess subscribes are ignored with a warning.
5. Shutdown cleanup — peer_topics and per-peer subscription counts cleared in Pubsub.run() finally (js topics.clear() parity).
6. Optional allowed_topics allowlist — remote subscribes outside the set are ignored (js allowedTopics / go allowlist filter parity).

Tests — tests/core/pubsub/test_pubsub_subscription_limits.py (12 cases), plus updates to test_handle_subscription and test_peers_subscribe. Coverage includes:

• Unsubscribe / disconnect / _handle_dead_peer / blacklist teardown cleanup
• Per-RPC and per-peer limits (reject + at-limit accept)
• Oversized inbound frames
• Idempotent resubscribe counting
• allowed_topics accept/reject/unsubscribe

CI — make lint, make typecheck, make test (2893 passed), make linux-docs all green locally.

What’s still missing / out of scope for this PR

Gap	Notes
PoC-scale regression test	No js-style flood test (~349k unique topics / heap measurement). Current tests use small N; sufficient for CI regression, not a full advisory reproduction.
Full resource manager	#572 remains open; this PR does not implement rcmgr-style global limits.
Score / blacklist on violation	Subscription floods are logged and dropped; no GossipSub scorer penalty or automatic blacklist (deferred for interoperability).
Inbound frame size vs js	py-libp2p caps pubsub RPC at 1 MiB; js advisory references 4 MiB LP frames — still blocks single-frame mega-RPC, but not identical.
Rich subscription filter API	allowed_topics covers the allowlist case; no full go-style SubscriptionFilter plugin interface.

Ecosystem parity (rough)

• go-libp2p-pubsub Implement validation throttler for message validation in Pubsub #710: matched + stronger default limits baked in
• js-libp2p 15.0.23: ~95% — core three defects addressed; PoC test and 4 MiB LP default differ

Happy to adjust default limits, add scorer penalties, or scope a follow-up PoC test if maintainers prefer. Thanks for any guidance.