filestream: match globs only against directory entries that can match

[orestisfl]

Proposed commit message

With the shared directory-listing cache in place (#53136), a CPU profile
of a scan round over 400 inputs watching one directory of 500 files put
81% of the time in filepath.Match: every input still matched its glob
against every entry on every tick.

Compile each glob once in buildWalkGroups into its components, its
scan-order index, and the literal prefix and suffix of its last
component. A leaf entry is then matched only if it starts and ends with
those literals, and the sorted listing is narrowed by binary search to
the block sharing the prefix: <pod>_<namespace>_<container>-*.log gains
through its prefix, *-<container-id>.log through its suffix.

The saving is per input, so it holds however many inputs share a
directory and grows with directory size; a glob that is only wildcards
gains nothing.

Matching semantics are unchanged: the prefilter only skips names no
pattern could match, so the sequence of process calls is identical,
which a property test checks against filepath.Match on random listings
and patterns.

Components are validated at construction with path.Match, which parses
the whole component whatever the name; filepath.Match stops after a
failed chunk, so a bad token behind a '*' used to be reported once per
scan, and only when an entry reached it. Malformed patterns are now
logged once and dropped, and the per-scan error handling is gone.

Interleaved runs against main, 500 files:
                                                         │      main      │                branch                │
                                                         │     sec/op     │    sec/op      vs base               │
GetFilesSharedDir/glob=prefix/cached=false/inputs=10-14      783.2µ ± 44%   493.2µ ±   3%  -37.03% (p=0.002 n=6)
GetFilesSharedDir/glob=prefix/cached=false/inputs=100-14     7.959m ±  4%   5.077m ±   2%  -36.21% (p=0.002 n=6)
GetFilesSharedDir/glob=prefix/cached=false/inputs=400-14     32.60m ±  4%   21.37m ±   5%  -34.46% (p=0.002 n=6)
GetFilesSharedDir/glob=prefix/cached=true/inputs=10-14      287.32µ ±  1%   19.62µ ±   3%  -93.17% (p=0.002 n=6)
GetFilesSharedDir/glob=prefix/cached=true/inputs=100-14     2878.2µ ±  1%   203.0µ ±   3%  -92.95% (p=0.002 n=6)
GetFilesSharedDir/glob=prefix/cached=true/inputs=400-14    11722.1µ ± 31%   873.4µ ±  44%  -92.55% (p=0.002 n=6)
GetFilesSharedDir/glob=suffix/cached=false/inputs=10-14     4970.6µ ± 16%   534.1µ ±   2%  -89.25% (p=0.002 n=6)
GetFilesSharedDir/glob=suffix/cached=false/inputs=100-14    49.906m ±  3%   5.410m ±   3%  -89.16% (p=0.002 n=6)
GetFilesSharedDir/glob=suffix/cached=false/inputs=400-14    201.57m ±  3%   22.53m ±  25%  -88.82% (p=0.002 n=6)
GetFilesSharedDir/glob=suffix/cached=true/inputs=10-14     4455.06µ ± 17%   58.14µ ±  23%  -98.69% (p=0.002 n=6)
GetFilesSharedDir/glob=suffix/cached=true/inputs=100-14    44415.0µ ± 42%   594.0µ ±  32%  -98.66% (p=0.002 n=6)
GetFilesSharedDir/glob=suffix/cached=true/inputs=400-14    176.116m ±  1%   2.387m ± 140%  -98.64% (p=0.002 n=6)
geomean                                                      11.08m         999.9µ         -90.98%


Benchmarks without literal prefix:
                               │     main     │                branch                │
                               │    sec/op    │    sec/op      vs base               │
GetFilesSelective-14             48.31m ± 13%    40.88m ± 11%  -15.37% (p=0.002 n=6)
GetFilesExcludeMost-14           99.51m ±  8%   102.82m ±  8%        ~ (p=0.132 n=6)
GetFilesLiteralMidComponent-14   16.98m ±  9%    16.90m ± 10%        ~ (p=1.000 n=6)
GetFilesMixed-14                 215.6m ± 14%    200.5m ±  9%        ~ (p=0.240 n=6)
geomean                          64.77m          61.44m         -5.14%

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

Malformed glob patterns are rejected at input start. A pattern such as /var/log/app*[ used to log glob match("...") failed: syntax error in pattern once per scan, and only when an entry reached the bad token. It is now dropped with one invalid glob pattern "...": syntax error in pattern at construction. No such pattern could ever match a file, so ingestion is unaffected; only the log line and its frequency change.

How to test this PR locally

go test ./filebeat/input/filestream/... -count=1 -race
go test ./filebeat/input/filestream/ -count=1 -tags integration
go test ./filebeat/input/filestream/ -run XXX -bench BenchmarkGetFilesSharedDir -count 6 > new.txt && benchstat -col /cached new.txt

Related issues

• Relates filestream: cache directory listings across inputs sharing the same base directory #53130
• Follow-up to filestream: cache directory listings across inputs sharing the same base directory #53136
• Relates filestream: single-pass file scanning; fix data duplication under file descriptor exhaustion #51694

Diagrams

Why: the leaf-matching hot path, main vs this branch

flowchart LR
    classDef hot fill:#ffe0e0,stroke:#c0392b,color:#7b241c
    classDef cheap fill:#e3f6e8,stroke:#1e8449,color:#145a32
    classDef note fill:#fdf6e3,stroke:#b7950b,color:#7d6608

    subgraph BEFORE["(before) — every input matches every entry"]
        direction TB
        B1["scan tick<br/>400 inputs x 1 shared directory"]
        B2["dirCache serves the listing once<br/>(#53136) — 500 names, sorted"]
        B3["walk: flatten walkGroup.byDepth<br/>into []walkPattern on every scan"]
        B4["for every name in the 500 entries"]
        B5["filepath.Match(comp, name)<br/>full glob engine per name"]
        B6["match? -> process(path, orderIndex)"]
        B7["profile: 81% of scan CPU in filepath.Match<br/>400 x 500 = 200k Match calls per tick"]
        B1 --> B2 --> B3 --> B4 --> B5 --> B6
        B5 -.-> B7
    end

    subgraph AFTER["(after) — only entries that can match are matched"]
        direction TB
        A1["scan tick<br/>400 inputs x 1 shared directory"]
        A2["dirCache serves the listing once<br/>(#53136) — 500 names, sorted"]
        A3["walkGroup.patterns compiled once in<br/>buildWalkGroups: comps, leafPrefix,<br/>leafSuffix, orderIndex"]
        A4["leafCandidates(names, exact)<br/>binary search on leafPrefix<br/>narrows the sorted slice"]
        A5["HasPrefix(name, leafPrefix) and<br/>HasSuffix(name, leafSuffix)<br/>two string compares"]
        A6["filepath.Match(comp, name)<br/>only for survivors"]
        A7["match? -> process(path, orderIndex)<br/>first pattern wins, break"]
        A1 --> A2 --> A3 --> A4 --> A5 --> A6 --> A7
        A5 -- "fails a literal end" --> A8["skipped, no Match call"]
        A4 -- "span empty" --> A9["whole directory skipped"]
    end

    BEFORE ==> AFTER

    class B4,B5,B7 hot
    class A4,A5,A8,A9 cheap
    class A3,A6 note

Loading

Setup: buildWalkGroups compiles and validates each glob once, not per scan

flowchart TB
    classDef added fill:#e3f6e8,stroke:#1e8449,color:#145a32
    classDef removed fill:#ffe0e0,stroke:#c0392b,color:#7b241c

    START(["newFileScanner -> buildWalkGroups()"]) --> IDX

    IDX["build s.pathIndex: path -> configured position<br/>s.pathsCanOverlap = pathsCanOverlap(s.paths)<br/>moved to the top: orderIndex is needed while compiling"]:::added
    IDX --> LOOP{"for each path in s.paths"}

    LOOP -- "no glob metacharacter" --> LIT["s.literals = append(...)<br/>resolved by stat, never walked"]
    LOOP -- "has glob metacharacter" --> ROOT["root = globRoot(path)<br/>longest literal leading directory"]

    ROOT --> NWP["newWalkPattern(root, path, s.pathIndex[path])"]:::added

    subgraph NEW["newWalkPattern (new)"]
        direction TB
        N1["comps = patternComponents(root, path)"]
        N2{"len(comps) == 0 ?"}
        N3["error: no path component below base directory"]
        N4["validate every component c with<br/>path.Match(c, #quot;#quot;)<br/>path.Match parses the whole component<br/>whatever the name; filepath.Match stops<br/>after a failed chunk"]
        N5["ErrBadPattern"]
        N6["leaf = comps[len-1]<br/>leafPrefix = literalPrefix(leaf)<br/>leafSuffix = literalSuffix(leaf)"]
        N7["walkPattern{pattern, comps,<br/>leafPrefix, leafSuffix, orderIndex}"]
        N1 --> N2
        N2 -- yes --> N3
        N2 -- no --> N4
        N4 -- invalid --> N5
        N4 -- valid --> N6 --> N7
    end

    NWP --> N1
    N3 --> ERR
    N5 --> ERR
    ERR["log.Errorf(#quot;invalid glob pattern %q#quot;)<br/>pattern dropped — it can match nothing"]:::added
    N7 --> GRP["groups[root].patterns = append(..., wp)"]

    GRP --> SORT["per group: SortStableFunc by len(comps)<br/>ascending depth, then configured order<br/>replaces the byDepth map + per-walk flatten"]:::added
    SORT --> OUT(["s.walkGroups, s.literals ready<br/>walk() just recurses over g.patterns"])

    subgraph GONE["removed from the scan path"]
        direction TB
        R1["walkGroup.byDepth map[int][]string + maxDepth"]:::removed
        R2["per-walk flattening of byDepth into []walkPattern<br/>(re-split components on every scan)"]:::removed
        R3["badPatterns dedup map + logBadPattern closure<br/>(ErrBadPattern raised once per candidate name)"]:::removed
    end

    SORT -.-> GONE

Loading

How: leafCandidates + matchLeaf, and what each glob shape costs per scan

flowchart TB
    classDef step fill:#eef3fb,stroke:#2471a3,color:#1a5276
    classDef cheap fill:#e3f6e8,stroke:#1e8449,color:#145a32
    classDef hot fill:#ffe0e0,stroke:#c0392b,color:#7b241c

    N0["sorted listing from readNames / dirCache<br/>500 names<br/>pod-0000-container-0000.log ... pod-0499-container-0499.log"]:::step

    N0 --> LC["leafCandidates(names, exact)<br/>for every exact pattern p:<br/>start = slices.BinarySearch(names, p.leafPrefix)<br/>extend while HasPrefix(names[end], p.leafPrefix)<br/>lo, hi = enclosing span of every block"]:::cheap

    LC -- "some p.leafPrefix is empty" --> WIDE["span widens to the whole slice<br/>(returns names unchanged)"]:::step
    LC -- "lo >= hi" --> NONE["nil — no entry can match,<br/>directory costs zero Match calls"]:::cheap
    LC -- "span" --> SPAN["names[lo:hi] — same sorted order"]:::step

    WIDE --> SPAN
    SPAN --> ML["matchLeaf(name): for p in exact"]:::step

    ML --> P1{"strings.HasPrefix(name, p.leafPrefix)"}:::cheap
    P1 -- no --> SKIP["continue — next pattern"]:::cheap
    P1 -- yes --> P2{"strings.HasSuffix(name, p.leafSuffix)"}:::cheap
    P2 -- no --> SKIP
    P2 -- yes --> M["matchName(p.comps[childDepth-1], name)<br/>filepath.Match, error impossible:<br/>components validated in newWalkPattern"]:::hot
    M -- no --> SKIP
    M -- yes --> PR["process(filepath.Join(dir, name), p.orderIndex)<br/>break — first pattern wins (unchanged)"]:::step

    subgraph EX["what each glob shape costs per scan"]
        direction TB
        E1["prefix shape pod-0123-container-*.log<br/>leafPrefix = pod-0123-container-<br/>leafSuffix = .log<br/>binary search -> 1 candidate -> 1 Match"]:::cheap
        E2["suffix shape *-container-0123.log<br/>leafPrefix is empty (no narrowing)<br/>leafSuffix = -container-0123.log<br/>500 HasSuffix -> 1 Match"]:::cheap
        E3["main — either shape<br/>500 filepath.Match calls"]:::hot
    end

Loading

Where it plugs in: walk()'s recursion, and the invariants that keep behaviour identical

flowchart TB
    classDef changed fill:#e3f6e8,stroke:#1e8449,color:#145a32
    classDef same fill:#eef3fb,stroke:#2471a3,color:#1a5276
    classDef inv fill:#fdf6e3,stroke:#b7950b,color:#7d6608

    W(["walk(g, process, recordUnobservable)"]) --> REC["rec(g.root, 0, g.patterns)<br/>was: rec(root, 0, patterns) built by<br/>flattening byDepth on every scan"]:::changed

    REC --> SPLIT["childDepth = depth + 1<br/>partition alive into:<br/>exact  = len(p.comps) == childDepth<br/>deeper = len(p.comps) #gt; childDepth"]:::same

    SPLIT --> ISLEAF{"len(deeper) == 0 ?"}

    ISLEAF -- yes --> RN["readNames(dir, shared)<br/>Readdirnames + sort, no DirEntry allocation<br/>(dirCache when shared)"]:::same
    RN --> CAND["leafCandidates(names, exact)<br/>binary-search narrowing"]:::changed
    CAND --> ML1["matchLeaf(name) for each candidate"]:::changed

    ISLEAF -- no --> RE["readEntries(dir, shared)<br/>DirEntry needed for IsDir / IsSymlink"]:::same
    RE --> LOOPE["for each entry e"]:::same
    LOOPE --> ML2["matchLeaf(e.Name())"]:::changed
    LOOPE --> DIRQ{"e is dir or symlink ?"}
    DIRQ -- no --> NEXT["next entry"]:::same
    DIRQ -- yes --> CHILD["childAlive = deeper patterns whose<br/>matchName(p.comps[childDepth-1], e.Name()) is true<br/>was: filepath.Match + logBadPattern on error"]:::changed
    CHILD -- "empty" --> PRUNE["prune: nothing below can match"]:::same
    CHILD -- "non-empty" --> STAT["symlink -> os.Stat to resolve"]:::same
    STAT --> RECD["rec(full, childDepth, childAlive)<br/>depth bounded by pattern depth:<br/>RecursiveGlobDepth cap kept,<br/>symlink cycles stay safe"]:::same
    RECD --> SPLIT

    ERR["read error -> isObservationError ?<br/>yes: recordUnobservable(dir)<br/>always: log.Debugf"]:::same
    RN -. error .-> ERR
    RE -. error .-> ERR

    subgraph INV["semantics deliberately unchanged"]
        direction TB
        I1["a name failing a literal end cannot match,<br/>so the prefilter drops no match"]:::inv
        I2["the narrowed span is iterated in the same sorted order"]:::inv
        I3["matchLeaf still breaks on the first matching pattern,<br/>with the same p.orderIndex"]:::inv
        I4["=> identical sequence of process() calls, so the winning<br/>path on a file-identity collision (matchedEarlier) is identical"]:::inv
        I1 --> I4
        I2 --> I4
        I3 --> I4
    end

    ML1 -.-> INV
    ML2 -.-> INV

Loading

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[Copilot]

🟢 Approval recommended

The optimization preserves matching behavior, validates malformed patterns early, and has focused correctness and performance coverage.

Pull request overview

Optimizes filestream glob scanning by precomputing pattern metadata and filtering directory entries before invoking filepath.Match.

Changes:

• Compiles, validates, and orders walk patterns during scanner construction.
• Narrows leaf candidates using literal prefixes and suffixes.
• Adds correctness tests, malformed-pattern coverage, benchmarks, and a changelog entry.

File summaries

File	Description
filebeat/input/filestream/fscanner.go	Implements compiled patterns and leaf prefiltering.
filebeat/input/filestream/fswatch_test.go	Tests validation, ordering, and filtering correctness.
filebeat/input/filestream/fswatch_bench_test.go	Benchmarks prefix/suffix patterns with and without caching.
changelog/fragments/1789144635-filestream-faster-glob-scans.yaml	Documents the performance enhancement.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filestream-glob-literal-prefilter upstream/filestream-glob-literal-prefilter
git merge upstream/main
git push upstream filestream-glob-literal-prefilter

[mergify]

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.