[medium] fix: csv feed with header=true imports zero attributes

[elhoim]

BLUF

• Priority: medium
• Bug: in fetch_csv_feed, the header-skip branch continues before the index += 1 at the bottom of the loop body, so index never leaves 0.
• Impact: every CSV feed with csvConfig.header true skips all its rows and imports nothing — run after run, while still reporting "result": "success".
• Fix: drive the row counter with enumerate(rows) and drop the manual increment, so the guard matches the first row only.
• Scope: 3 lines in api/app/worker/tasks.py, plus one new mock-only test module. No API schema change, so docs/features/api/openapi.json is untouched.

What was wrong

api/app/worker/tasks.py initialises index = 0 before the ingestion loop and increments it as the last statement of the loop body:

for row in rows:
    if db_feed.settings["csvConfig"]["header"] and index == 0:
        continue  # skip the first line if header is present
    ...
    index += 1

When csvConfig.header is true, the guard fires on the first iteration and continue jumps straight back to the for, never reaching index += 1. index therefore stays 0 for every subsequent row and the guard fires forever.

The header is not stripped anywhere upstream either: feeds_repository.parse_csv_feed_lines simply runs csv.reader over the lines and returns every parsed row, so this guard is the only header handling in the CSV ingestion path.

Concrete impact

A CSV feed configured with a header line imports zero attributes on every fetch. Because the exceptions counter is never reached either, the task still returns:

{"result": "success",
 "message": "CSV feed=<name> processed, 0 rows parsed, 0 attributes created, 0 rows failed."}

There is no error, no failed row and no log line — the feed just looks empty, indefinitely.

What this changes

• for row in rows: becomes for index, row in enumerate(rows):.
• The manual index = 0 initialiser and the trailing index += 1 are removed; enumerate now owns the counter.

Nothing else in the function reads index, and feeds without a header behave exactly as before.

How it was verified

• New regression test api/app/tests/services/test_feed_worker_tasks.py, following the mock-based style of the existing test_correlation_worker_tasks.py (no service containers needed):
  • test_skips_the_header_row_only — 3 CSV lines with header=true. On unpatched main it reports 0 rows parsed, 0 attributes created; with the fix it reports 2 rows parsed, 2 attributes created, 0 rows failed and asserts process_csv_feed_row is called with the two data rows and never with the header row.
  • test_keeps_every_row_when_there_is_no_header — same feed with header=false, asserting all 3 rows are still ingested, so a future fix cannot over-skip.
• Run with cd api && poetry run pytest app/tests/services/test_feed_worker_tasks.py.
• Whole-tree Python syntax check clean.

---

Local verification

python3 - <<'PY'
import ast, csv, sys
from unittest.mock import MagicMock

def grab(path, name):
    src = open(path).read()
    tree = ast.parse(src)
    for n in ast.walk(tree):
        if isinstance(n, ast.FunctionDef) and n.name == name:
            return ast.get_source_segment(src, n)
    raise SystemExit("function %s not found in %s" % (name, path))

ns = {"csv": csv}
exec(grab("api/app/repositories/feeds.py", "parse_csv_feed_lines"), ns)

feeds = MagicMock()
feeds.parse_csv_feed_lines = ns["parse_csv_feed_lines"]
feeds.fetch_csv_content_from_local.return_value = ["value", "1.2.3.4", "5.6.7.8"]
feeds.process_csv_feed_row.side_effect = lambda row, settings: {"type": "ip-src", "value": row[0]}

db_feed = MagicMock()
db_feed.settings = {"csvConfig": {"header": True, "delimiter": ","}}
db_feed.input_source = "local"
db_feed.name = "csv-feed"
feeds.get_feed_by_id.return_value = db_feed

g = {
    "logger": MagicMock(), "Session": MagicMock(), "engine": MagicMock(),
    "users_repository": MagicMock(), "feeds_repository": feeds,
    "attributes_repository": MagicMock(), "events_repository": MagicMock(),
    "attribute_schemas": MagicMock(),
}
exec(grab("api/app/worker/tasks.py", "fetch_csv_feed"), g)
res = g["fetch_csv_feed"](1, 1)
print("HEADER=TRUE ->", res["message"])
rows = [c.args[0] for c in feeds.process_csv_feed_row.call_args_list]
print("rows ingested:", rows)
assert res["message"] == ("CSV feed=csv-feed processed, 2 rows parsed, "
                         "2 attrib

Output:

Patched tree (exit 0):
HEADER=TRUE -> CSV feed=csv-feed processed, 2 rows parsed, 2 attributes created, 0 rows failed.
rows ingested: [['1.2.3.4'], ['5.6.7.8']]
HEADER=FALSE -> CSV feed=csv-feed processed, 3 rows parsed, 3 attributes created, 0 rows failed.
OK: header row skipped exactly once; no-header feed keeps all rows

Baseline (same heredoc, patch stashed, exit 1):
HEADER=TRUE -> CSV feed=csv-feed processed, 0 rows parsed, 0 attributes created, 0 rows failed.
rows ingested: []
AssertionError: HEADER SKIP BUG PRESENT

• Verified on a scratch work tree with the patch applied.
• The same check was re-run with the change stashed and fails on unpatched code, so it discriminates rather than merely passing.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.56%. Comparing base (2d90051) to head (6f4fa23).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #389      +/-   ##
==========================================
+ Coverage   84.40%   84.56%   +0.16%     
==========================================
  Files         208      209       +1     
  Lines       19402    19425      +23     
==========================================
+ Hits        16376    16427      +51     
+ Misses       3026     2998      -28     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[righel]

Good catch, thanks!