Validate API result count matches totalEventCount

[JacobPEvans-personal]

Problem

The Cribl API returns a totalEventCount field indicating how many events should be in the result set. The current implementation tracks this value (line 319-320, 386-387 in bin/goatsearch.py) and separately counts yielded events via self.offset (line 416), but there's no verification that these values match upon completion.

If the counts diverge, it could indicate:

• Events were missed during pagination
• Duplicate events were yielded
• The API returned inconsistent data

Current Behavior

# Line 319-320: Tracks total from status endpoint
if 'totalEventCount' in job_status and job_status['totalEventCount'] > self.total_event_count:
    total_event_count = job_status['totalEventCount']

# Line 416: Increments for each yielded event
self.offset = self.offset + 1

No comparison is made between total_event_count and offset at completion.

Proposed Solution

1. Add parity check at job completion:

   if self.job_complete and self.offset != self.total_event_count:
       self.write_warning(f"Event count mismatch: expected {self.total_event_count}, yielded {self.offset}")

2. Add test cases (depends on Add end-to-end test infrastructure for CriblSearch #26):

   • Test that verifies yielded count equals totalEventCount for normal completion
   • Test with mocked partial/interrupted results
   • Test edge case where API reports 0 events

Acceptance Criteria

• Parity check implemented at job completion
• Warning logged when mismatch detected (non-blocking)
• Test cases covering normal and edge scenarios

Dependencies

• Depends on Add end-to-end test infrastructure for CriblSearch #26 (test infrastructure)

Related Code

• bin/goatsearch.py lines 319-320, 386-387, 416

[JacobPEvans-personal]

** Add events yielded count, and compare that to the total