Skip to content

Drop new logs once the queue holds maxQueueSize of them - #32

Merged
PetrHeinz merged 4 commits into
mainfrom
claude/max-queue-size
Sep 30, 2026
Merged

PetrHeinz merged 4 commits into
mainfrom
claude/max-queue-size

Conversation

@PetrHeinz

Copy link
Copy Markdown
Member

Drain queues logs without limit. While the endpoint is down, a busy application keeps growing (the red-team run measured about 600 MB RSS at 300k queued logs) until it is killed, and then everything queued is lost anyway.

This bounds the queue the way our Java client does with maxQueueSize:

  • New target option maxQueueSize, 100000 by default. Once the queue holds that many logs, new logs are dropped; nothing already queued is removed.
  • The first dropped log writes one error to NLog's internal log: "maximum number of logs in the queue reached (N). New logs will be dropped." It is written again for the next overflow once the drain has taken logs off the queue, so a later overflow is reported too, but a queue that stays full does not flood the internal log.
  • The queue length is an Interlocked counter kept on enqueue and dequeue, not ConcurrentQueue.Count, which walks the segments.
  • maxQueueSize below 1 is reported as NLogConfigurationException. The check sits at the top of InitializeTarget rather than next to the checks of Validate sourceToken and endpoint, accept a bare ingesting host #22 and Reject maxBatchSize and flushPeriodMilliseconds below 1 and negative retries #28, so that this PR merges with them without conflicts.
  • Drain is public API: its constructor keeps its signature, and an overload takes maxQueueSize. The existing constructor uses the same default of 100000.
  • The example README mentions the option under "Additional configuration" and in the snippet.

This changes behaviour for existing users: the queue used to be unlimited, now it holds at most 100000 logs by default. An application that logs faster than the endpoint takes its logs for long enough now loses the newest logs instead of growing until it runs out of memory.

The first commit adds the tests and does not compile, since they use the new option. With only the property added (checked locally), the 8 logs written while a batch waits for its retry are all queued and delivered, nothing reaches the internal log, and maxQueueSize="0" is accepted. The second commit bounds the queue.

The tests restore NLog's internal log settings in Dispose with the same lines #20 adds, so the two merge cleanly in either order.

🤖 Generated with Claude Code

PetrHeinz and others added 2 commits September 30, 2026 17:19
The tests use the new maxQueueSize option, so this commit does not
compile yet. With only the property added, all 8 logs written while a
batch is stuck are queued and delivered, nothing is reported, and
maxQueueSize="0" is accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The drain queued without limit, so an application whose endpoint is
down grew until it was killed. New target option maxQueueSize (100000
by default) bounds the queue like the Java client does: a full queue
drops new logs and reports the first one in NLog's internal log, again
for the next overflow once the drain has taken logs off the queue. The
length is an Interlocked counter, since ConcurrentQueue.Count walks
the segments. maxQueueSize below 1 is a configuration error. Drain
keeps its constructor and gets an overload that takes the limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PetrHeinz
PetrHeinz marked this pull request as ready for review September 30, 2026 15:37
@PetrHeinz
PetrHeinz merged commit 0791554 into main Sep 30, 2026
16 checks passed
@PetrHeinz
PetrHeinz deleted the claude/max-queue-size branch September 30, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant