Experiment/search text by xuy8w89 · Pull Request #212 · cocoindex-io/cocoindex-code

xuy8w89 · 2026-07-13T15:40:51Z

Summary

Adds ccc search --text — literal + regex full-text search, alongside semantic search and structural grep. Addresses #192.

It's an index-free local scan: like grep, it branches before the daemon/project gate (runs in any directory, no ccc init needed) and reuses file_walk, so the searched file set + whitelist match the indexer. Scope and design were aligned with @georgeh0 in the thread.

Query model

Terms are AND-combined — a file matches only if it contains every term.
/regex/ marks a regex, otherwise literal — GitHub-style: /…/ is a regex only when its body has no free (unescaped) /, so /blobs/v1/ stays literal.
Smart-case by default (-i / -s to force); reuses --lang / --path; --limit caps results; --refresh is rejected (index-only).
Covers docs/config that grep skips (any included file, not just code-matchable languages).

Notes

Inherits the indexer whitelist (Dockerfile, .github/**, dotfiles excluded — an opt-in --all-files can come later).
35 tests (tests/test_textsearch.py); ruff / mypy clean. Scan is linear (~0.11 ms/file) — cocoindex (859 files) runs a query in ~0.36 s.

Next steps — optional full-text index

Non-indexed by design for v1 (per @georgeh0, indexing can be a future opt-in). A live scan stays fast for typical repos but grows linearly, so very large trees (100K–1M files) will want an index.

Planned follow-up: an opt-in inverted index — likely SQLite FTS5, maintained incrementally by the daemon like the vector index — that search --text queries for literal terms, falling back to the live scan for regex and for always-fresh results. This PR is already shaped for it: matching / rendering / walk are index-agnostic, and the new Term type carries the per-term literal-vs-regex signal the index needs (literals → index lookup, regex → scan).

A lexical search path — literal terms + regex, GitHub-code-search style — alongside semantic `search` and structural `grep`. Runs entirely locally (no daemon, no index, no project init): `--text` branches before the project/daemon gate like `grep`, and reuses `file_walk` so the searched file set matches the indexer. - File-level AND across terms; `/re/` marks a regex, `\/` escapes a literal; smart-case by default with `-i` / `-s` overrides; reuses `--lang` / `--path`. - Rejects `--refresh` in `--text` mode (index-only). - Unlike `grep`, covers non-code files (docs/config) with no code language. Experimental, scoped to issue cocoindex-io#192. `--limit` / `--offset` and default-path behavior are pending maintainer confirmation. Adds tests/test_textsearch.py (33 tests).

- compile_terms now yields Term{raw, is_regex, case_sensitive, pattern}, giving a future inverted index a per-term literal-vs-regex signal instead of a bare compiled pattern. - A /…/ term is a regex only when its body has no free (unescaped) slash, matching GitHub code search (e.g. /blobs/docs-v1/ stays literal); drops the \/ escaping-outside-regex rule. - Honor --limit in --text mode (caps printed files), per maintainer feedback. All contained in textsearch.py + its tests; grep and semantic search untouched.

Copilot

Pull request overview

Adds a new ccc search --text mode that performs local, index-free full-text search (literal substrings and /regex/ terms) over the same include/exclude + .gitignore file set used by the indexer/ccc grep, with streaming per-file output.

Changes:

Introduces cocoindex_code.textsearch engine: term parsing (smart-case, literal vs regex), file walking, matching, and rendering.
Extends ccc search CLI with --text plus case/color controls, branching before the daemon/project gate.
Adds an end-to-end test suite covering parsing, matching semantics, filters, rendering, and CLI behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
`tests/test_textsearch.py`	Adds comprehensive tests for term parsing, engine behavior, rendering, and CLI `search --text`.
`src/cocoindex_code/textsearch.py`	Implements the local text search engine (term compilation, file iteration, matching, and rendering).
`src/cocoindex_code/cli.py`	Adds `ccc search --text` entrypoint and streaming output logic with case/color options.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

xuy8w89 · 2026-07-15T14:02:23Z

+    if text:
+        if refresh:
+            # --refresh rebuilds the semantic index — meaningless for a live scan.
+            _typer.echo(
+                "Error: --refresh does not apply to --text (live scan, no index).", err=True
+            )
+            raise _typer.Exit(code=1)
+        _run_text_search(


Done — --text now errors on --offset, same as --refresh: both are index concepts with no meaning for a live scan. (1a1c7c0)

xuy8w89 · 2026-07-15T14:02:37Z

+        return
+
    project_root = str(require_project_root())


Done — --ignore-case / --no-color now error when used without --text, instead of being silently ignored. (--case-sensitive was removed entirely — see @georgeh0's note below.) (1a1c7c0)

georgeh0 · 2026-07-14T05:05:41Z

+    ignore_case: bool = _typer.Option(
+        False, "-i", "--ignore-case", help="[--text] Force case-insensitive matching."
+    ),
+    case_sensitive: bool = _typer.Option(
+        False, "-s", "--case-sensitive", help="[--text] Force case-sensitive matching."
+    ),


They're redundant to each other. We only need one.

Dropped -s/--case-sensitive and kept just -i/--ignore-case. Smart-case already gives case-sensitivity when a term has an uppercase letter, so the only override worth a flag is force-insensitive (-i). (1a1c7c0)

georgeh0 · 2026-07-14T05:43:09Z

+            return
+        block = _ts.render_file(item, color=use_color)  # render outside the lock
+        with output_lock:
+            if limit and matched >= limit:


Ideally when the limit is reached, we should stop scanning the files at higher level.

Got it, I'll fix that

Done — the cap now lives in the engine: TextSearch.run takes a limit and stops feeding the walk once that many files have matched, rather than scanning everything and capping the output. (e6476c8)

georgeh0 · 2026-07-14T05:46:05Z

+
+    if matched == 0:
+        _typer.echo("No matches found.")
+


I think if we stopped after hitting limit, we may output a message to hint about this, and may say something like if want to see more output, retry with a higher limit.

Added — when it stops at the limit it prints Stopped at --limit N. Re-run with a higher --limit to see more. (run returns whether it stopped early, so the CLI knows when to show it.) (e6476c8)

georgeh0 · 2026-07-14T05:48:32Z

+            # --refresh rebuilds the semantic index — meaningless for a live scan.
+            _typer.echo(
+                "Error: --refresh does not apply to --text (live scan, no index).", err=True
+            )


--offset is also not supported for --text, so we should also echo an error if offset is not 0.

Done — --text now errors on --offset, same as --refresh: both are index concepts with no meaning for a live scan. (1a1c7c0) (same as the reply for copilot)

xuy8w89 · 2026-07-14T11:11:59Z

Thank you @georgeh0 , I'll fix these by tmr.

@georgeh0

…rop -s - --text now rejects --offset (like --refresh): both are index concepts with no meaning for a live scan. - --ignore-case / --no-color error when used without --text instead of being silently ignored in semantic mode. - Drop the redundant -s / --case-sensitive; smart-case already covers the common case (uppercase → sensitive) and -i handles the force-insensitive override. Addresses review feedback on cocoindex-io#212 (@georgeh0, Copilot).

@georgeh0

Both `ccc grep` and `ccc search --text` duplicated the project file walk (resolve root → include/exclude + .gitignore matcher → path-glob → display paths). Factor it into `iter_project_files`, returning a `ProjectFiles` (files iterator + extension→language overrides). Each caller keeps its own per-file gating: grep skips files with no matchable code language; text search keeps every included file. Behavior-preserving — grep's tests stay green. Per review feedback on cocoindex-io#212 (@georgeh0).

@georgeh0

TextSearch.run takes an optional `limit`: it stops feeding the walk once that many files have matched (rather than only capping printed output) and returns whether it stopped early. `ccc search --text` uses that to print a "re-run with a higher --limit" hint when results were truncated. Addresses review feedback on cocoindex-io#212 (@georgeh0).

georgeh0

Mostly looks good.

Please also resolve the potential merge conflicts. Thanks!

georgeh0 · 2026-07-20T00:38:40Z

+        gutter = _paint(f"{lm.line_no:>{width}}| ", color, fg="bright_black")
+        parts.append(f"{gutter}{_highlight_line(lm.text, lm.spans, color)}")


The format is shared with grep.py. Is there any code worth sharing too?

Shared what's genuinely common: paint / path_header / gutter / gutter_width now live in a small render.py that both use — the header and <line>| gutter format are identical, so that's one definition now.

What stays per-search is the inside of a line: grep dims the context around a structural match (char offsets over a CodeMatch), while text search highlights the matched spans on a line — different data and different emphasis. Output is unchanged; the existing render tests pass as-is. (6787904)

Resolved src/cocoindex_code/cli.py: both sides added new `search` options, so keep both — upstream's `--json` alongside this branch's `--text` / `-i` / `--no-color`. `--json` has no defined shape for line-level text matches yet, so `--text --json` now errors instead of silently ignoring the flag (consistent with how `--refresh` and `--offset` are handled in text mode).

@georgeh0

… --text `grep` and `search --text` print the same shape — a bold path header, then `<line>| <text>` behind a dim gutter — and were duplicating those styling primitives. Moved `paint` / `path_header` / `gutter` / `gutter_width` into a small `render.py`; both now use it. What each search draws *inside* a line stays where it was: grep dims the context around a structural match, text search highlights the matched spans. Output is unchanged — the existing render tests pass as-is. Per review feedback on cocoindex-io#212 (@georgeh0).

xuy8w89 · 2026-07-22T15:23:31Z

Mostly looks good.

Please also resolve the potential merge conflicts. Thanks!

Merged upstream/main and resolved the conflict — it was just both sides adding new search options, so I kept both (--json from main alongside --text / -i / --no-color).
One thing worth flagging: --json has no defined shape for line-level text matches, so --text --json now errors instead of silently ignoring the flag. Happy to implement JSON output for text results if you'd like it in this PR.

xuy8w89 added 2 commits July 5, 2026 23:07

badmonster0 requested review from Copilot and georgeh0 July 13, 2026 22:32

Copilot started reviewing on behalf of badmonster0 July 13, 2026 22:33 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

georgeh0 reviewed Jul 14, 2026

View reviewed changes

xuy8w89 added 3 commits July 15, 2026 20:58

xuy8w89 requested a review from georgeh0 July 15, 2026 14:13

georgeh0 reviewed Jul 20, 2026

View reviewed changes

xuy8w89 requested a review from georgeh0 July 20, 2026 11:56

xuy8w89 added 2 commits July 21, 2026 23:16

		gutter = _paint(f"{lm.line_no:>{width}}\| ", color, fg="bright_black")
		parts.append(f"{gutter}{_highlight_line(lm.text, lm.spans, color)}")

Uh oh!

Conversation

xuy8w89 commented Jul 13, 2026

Summary

Query model

Notes

Next steps — optional full-text index

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuy8w89 commented Jul 14, 2026

Uh oh!

georgeh0 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xuy8w89 commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants