loki: prevent duplicate log ingestion and improve timestamp handling#4498
loki: prevent duplicate log ingestion and improve timestamp handling#4498Anulo2 wants to merge 3 commits into
Conversation
…ling Remove the 'else if infinite' branch in updateURI() that reset the query start timestamp backward by 1 minute when no new results were returned. This caused the same Loki log entries to be re-fetched and ingested multiple times, inflating event counts and triggering false bucket overflows. Also improve robustness: - Scan all streams for maxTS instead of only the first stream's last entry - Use strconv.ParseInt/FormatInt for nanosecond timestamps to avoid 32-bit Atoi overflow on platforms where int is 32-bit - Align entry.go timestamp parsing with types.go (time.Unix(0, t)) Fixes duplicate log lines appearing 2-3x in CrowdSec alerts when using Loki datasource in streaming mode.
|
@Anulo2: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.
DetailsI am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository. |
|
@Anulo2: There are no area labels on this PR. You can add as many areas as you see fit.
DetailsI am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository. |
|
/kind fix |
|
/area agent |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4498 +/- ##
===========================================
+ Coverage 37.57% 64.00% +26.42%
===========================================
Files 460 478 +18
Lines 33569 34310 +741
===========================================
+ Hits 12614 21959 +9345
+ Misses 19595 10158 -9437
- Partials 1360 2193 +833
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3 participants
Fixes duplicate log lines appearing 2-3x in CrowdSec alerts when using Loki datasource in streaming mode.