Skip to content

feat(mantis-explorer): new student activity web app with accurate escalation detection#38

Merged
liamadale merged 8 commits into
mainfrom
dev
Apr 29, 2026
Merged

feat(mantis-explorer): new student activity web app with accurate escalation detection#38
liamadale merged 8 commits into
mainfrom
dev

Conversation

@liamadale
Copy link
Copy Markdown
Owner

Summary

  • Adds Mantis Explorer — a new Flask + HTMX web app for browsing student ticket activity by institution and individual student, with resizable slide panels, date-range filtering, and ECharts visualisations
  • Rewrites escalation detection in the Mantis normaliser to eliminate systematic false positives and false negatives, and surfaces the is_escalated flag throughout the explorer UI
  • Registers the new app in the hub (run_all.py) at /mantis-explorer

What changed

New app — apps/mantis_explorer/

  • Institution overview table with sortable columns, search, and stat chips
  • Per-institution drill-down with a student activity table
  • Student slide panel showing tickets created and tickets commented on, with escalation badges on escalated rows
  • Ticket detail slide panel (reuses the same rendering as mantis_web) — resizable via drag with localStorage persistence
  • ECharts bar, timeline, and escalation charts
  • Dark/light theme toggle, HTMX progress bar, date-range filters that persist in the URL

Escalation detection rewrite — src/mantis/mantis_search.py

The previous single regex (\bescalat(e|ed|ing|ion)\b) produced both false positives and false negatives:

  • False positives: "no need to escalate", "privilege escalation", "worth escalating [in future]", conditional/future phrasing
  • False negatives: client was contacted using plain language ("I've informed the client", "Message to client:", "Customer communication:") without using the word "escalate"

Replaced with a two-stage function _note_is_escalation():

  1. Client-contact phrases (past/present tense only — future-tense occurrences preceded by "will" are skipped)
  2. Past-tense "escalated [this/it] to [the] client" (negated occurrences preceded by "not"/"won't" are skipped)

Validated against a labelled set of 10 tickets (5 true positives, 5 true negatives — all correct). Corpus impact: ~327 → ~410 escalated tickets after index rebuild.

Supporting changes

  • src/mantis/activity_report.py_ticket_ref() now includes is_escalated so it propagates into StudentStats.created_tickets
  • apps/mantis_explorer/templates/partials/student_panel.html — red exclamation icon and row tint on escalated tickets
  • apps/mantis_explorer/templates/partials/ticket_detail.html — red "Escalated" badge in the ticket meta row

Test plan

  • Run uv run python src/mantis/mantis_index.py to rebuild the ticket index with the new escalation logic
  • Start the hub (uv run python run_all.py) and navigate to /mantis-explorer
  • Verify institution table loads and filters work
  • Open an institution, click a student — confirm slide panel opens with correct escalation badges
  • Click an escalated ticket in the panel — confirm red "Escalated" badge appears in the ticket detail viewer
  • Confirm tickets previously mis-classified (e.g. #12771, #13009) are no longer marked escalated after index rebuild
  • Confirm tickets previously missed (e.g. #13539, #13665, #12723) are now marked escalated after index rebuild
  • Run uv run pytest — all tests pass

… search bar

Extend the query backend and web form to support filtering by destination
IP alongside the existing src_ip filter.

- build_base_query gains a dest_ip_filter param that appends a
  destination.ip term clause when set
- run_query guards dest_ip_filter the same way src_ip is guarded:
  only applied to modules that declare destination.ip in SOURCE_FIELDS
- build_search_params_from_request reads the dest_ip form value
- base.html search bar exposes a Dst IP text input
Zeek modules emit "—" (U+2014) as src_ip when no source IP is present
in a record. run_cross_protocol_query was aggregating these into a
phantom "—" row in the cross-protocol overview matrix.

Extend the IP guard to also skip em-dash values alongside empty strings.
…and fix sticky-column rendering

Sidebar navigation:
- Add always-visible Overview link (cross-protocol matrix) to sidebar
- Replace Hub home link with brand logo that routes to Hub when mounted
  under a script name, or to Overview when running standalone
- Polish collapsed sidebar: hide scrollbar, center icons, apply category
  colour accents to group headers, add .sidebar-overview highlight class
- Update category icons (alerts, network, web, remote, auth, messaging)
- Reorder MODULES so suricata_alert appears before weird in the registry

Filter persistence:
- Replace localStorage-based sensor/time_range persistence with
  sessionStorage so each browser tab maintains independent filter state
- Snapshot all filter keys (time_range, sensor, src_ip, dest_ip,
  direction, public_only, limit, min_risk) on form submit; restore on
  link navigation
- Remove inline onchange handler from the time_range select

CSS cache-busting:
- Compute pisces.css version from file mtime at app startup and inject
  as a ?v= query param in the stylesheet link

Sticky-column rendering:
- Raise col-ip-addr z-index to 20 (top-left corner) and add explicit
  top+left sticky declarations to thead .col-ip-addr and thead .col-total
  so both axes stick reliably at the corner
- Remove drop shadows from col-ip-addr and col-total; keep only the
  inset separator line
- Remove z-index from the shared thead rule to avoid clobbering the
  per-column corner values
Introduces djLint as a dev dependency and wires it into both pre-commit
and the GitHub Actions CI workflow, following the same advisory-on-dev /
blocking-on-PR-to-main pattern used by the existing ruff checks.

- djlint-jinja pre-commit hook lints Jinja templates on every commit
- CI: "Lint HTML (djlint check)" runs blocking on PRs to main,
  advisory on pushes to dev (mirrors ruff check behaviour)
- CI: "Format check HTML (djlint format)" runs advisory on all triggers
- [tool.djlint] config added to pyproject.toml: jinja profile,
  100-char line limit, H021/H023/H030/H031 suppressed with rationale
Resolves all djlint warnings to reach 0 errors across 32 HTML files:

- T003: added block names to all bare endblock tags (21 occurrences)
  across dashboard_web, mantis_web, and opensearch_web templates
- H006: added height/width attributes to brand logo img tags in 4
  base templates
- H025: added closing </option> tags to datalist options in
  filter_form.html
- H014: removed extra blank lines in opensearch_web base.html and
  record_detail.html
- T032: removed extra whitespace in Jinja set tags in ticket_detail.html
  and record_detail.html
- H029: lowercased form method="GET" to method="get" in opensearch_web
  base.html
- H020: replaced empty <span></span> with <span>&nbsp;</span> in
  threat_card.html

Also adds J018 to the djlint ignore list in pyproject.toml — cross-app
internal links cannot use url_for() in a multi-app Flask setup.
…n tickets

Replace the single loose _ESCALATION_RE pattern with a two-stage function
_note_is_escalation() that eliminates false positives:

- Stage 1: matches past-tense client-contact phrases (informed/notified the
  client, let the client know, reached out to the client, etc.) and skips
  any match whose 20-char prefix contains "will" (future intent).
- Stage 2: matches past-tense "escalated [this/it] to [the] client" and
  skips matches prefixed with "not" or "won't" (negated intent).

Previously, bare "escalat*" triggered on "privilege escalation", conditional
futures ("will let the client know"), and negations ("not going to escalate
to the client"). The two-stage approach targets only confirmed past-action
phrases.

_normalize_issue() now exposes is_escalated and escalated_by on every
normalised ticket dict. activity_report._ticket_ref() propagates is_escalated
into StudentStats.created_tickets. student_activity adds an "Escalated"
column to the summary table, shows a per-student count in the detail view,
supports --org / --since / --until CLI flags, and refactors the graph helper
into a reusable _plot_ticket_timeline() shared by per-student and per-org
views.
…loration

New app at /mantis-explorer (port 5003 standalone via apps/mantis_explorer/run.py)
that provides a browser-based view of the data from student_activity/activity_report.

Key capabilities:
- Institution overview table with ticket counts, escalation counts, and date ranges
- Per-institution student breakdown with sortable activity table
- Per-student slide panel showing created tickets (with escalated row tinting and
  red exclamation icon) and notes, plus a ticket detail view with an "Escalated"
  badge in the meta row when is_escalated is true
- Resizable ticket slide panel with drag-to-resize handle; width persisted to
  localStorage across page loads
- Charts: submission timeline, org bar chart, and escalation breakdown
- Date-range filtering propagated from the URL query string
- Dark SOC-analyst CSS theme consistent with the rest of the PISCES UI (me.css)
Mount the new mantis-explorer app at /mantis-explorer in the
DispatcherMiddleware and add a hub card linking to it with a
fa-user-graduate icon.

Also bumps locked dependency versions: rich 14→15, ruff 0.15.6→0.15.12,
pre-commit 4.5.1→4.6.0, geoip2 >=4.8.0→>=5.2.0, pytest >=9.0.3.
@liamadale liamadale merged commit e141f99 into main Apr 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant