Conversation
… search bar Extend the query backend and web form to support filtering by destination IP alongside the existing src_ip filter. - build_base_query gains a dest_ip_filter param that appends a destination.ip term clause when set - run_query guards dest_ip_filter the same way src_ip is guarded: only applied to modules that declare destination.ip in SOURCE_FIELDS - build_search_params_from_request reads the dest_ip form value - base.html search bar exposes a Dst IP text input
Zeek modules emit "—" (U+2014) as src_ip when no source IP is present in a record. run_cross_protocol_query was aggregating these into a phantom "—" row in the cross-protocol overview matrix. Extend the IP guard to also skip em-dash values alongside empty strings.
…and fix sticky-column rendering Sidebar navigation: - Add always-visible Overview link (cross-protocol matrix) to sidebar - Replace Hub home link with brand logo that routes to Hub when mounted under a script name, or to Overview when running standalone - Polish collapsed sidebar: hide scrollbar, center icons, apply category colour accents to group headers, add .sidebar-overview highlight class - Update category icons (alerts, network, web, remote, auth, messaging) - Reorder MODULES so suricata_alert appears before weird in the registry Filter persistence: - Replace localStorage-based sensor/time_range persistence with sessionStorage so each browser tab maintains independent filter state - Snapshot all filter keys (time_range, sensor, src_ip, dest_ip, direction, public_only, limit, min_risk) on form submit; restore on link navigation - Remove inline onchange handler from the time_range select CSS cache-busting: - Compute pisces.css version from file mtime at app startup and inject as a ?v= query param in the stylesheet link Sticky-column rendering: - Raise col-ip-addr z-index to 20 (top-left corner) and add explicit top+left sticky declarations to thead .col-ip-addr and thead .col-total so both axes stick reliably at the corner - Remove drop shadows from col-ip-addr and col-total; keep only the inset separator line - Remove z-index from the shared thead rule to avoid clobbering the per-column corner values
Introduces djLint as a dev dependency and wires it into both pre-commit and the GitHub Actions CI workflow, following the same advisory-on-dev / blocking-on-PR-to-main pattern used by the existing ruff checks. - djlint-jinja pre-commit hook lints Jinja templates on every commit - CI: "Lint HTML (djlint check)" runs blocking on PRs to main, advisory on pushes to dev (mirrors ruff check behaviour) - CI: "Format check HTML (djlint format)" runs advisory on all triggers - [tool.djlint] config added to pyproject.toml: jinja profile, 100-char line limit, H021/H023/H030/H031 suppressed with rationale
Resolves all djlint warnings to reach 0 errors across 32 HTML files: - T003: added block names to all bare endblock tags (21 occurrences) across dashboard_web, mantis_web, and opensearch_web templates - H006: added height/width attributes to brand logo img tags in 4 base templates - H025: added closing </option> tags to datalist options in filter_form.html - H014: removed extra blank lines in opensearch_web base.html and record_detail.html - T032: removed extra whitespace in Jinja set tags in ticket_detail.html and record_detail.html - H029: lowercased form method="GET" to method="get" in opensearch_web base.html - H020: replaced empty <span></span> with <span> </span> in threat_card.html Also adds J018 to the djlint ignore list in pyproject.toml — cross-app internal links cannot use url_for() in a multi-app Flask setup.
…n tickets
Replace the single loose _ESCALATION_RE pattern with a two-stage function
_note_is_escalation() that eliminates false positives:
- Stage 1: matches past-tense client-contact phrases (informed/notified the
client, let the client know, reached out to the client, etc.) and skips
any match whose 20-char prefix contains "will" (future intent).
- Stage 2: matches past-tense "escalated [this/it] to [the] client" and
skips matches prefixed with "not" or "won't" (negated intent).
Previously, bare "escalat*" triggered on "privilege escalation", conditional
futures ("will let the client know"), and negations ("not going to escalate
to the client"). The two-stage approach targets only confirmed past-action
phrases.
_normalize_issue() now exposes is_escalated and escalated_by on every
normalised ticket dict. activity_report._ticket_ref() propagates is_escalated
into StudentStats.created_tickets. student_activity adds an "Escalated"
column to the summary table, shows a per-student count in the detail view,
supports --org / --since / --until CLI flags, and refactors the graph helper
into a reusable _plot_ticket_timeline() shared by per-student and per-org
views.
…loration New app at /mantis-explorer (port 5003 standalone via apps/mantis_explorer/run.py) that provides a browser-based view of the data from student_activity/activity_report. Key capabilities: - Institution overview table with ticket counts, escalation counts, and date ranges - Per-institution student breakdown with sortable activity table - Per-student slide panel showing created tickets (with escalated row tinting and red exclamation icon) and notes, plus a ticket detail view with an "Escalated" badge in the meta row when is_escalated is true - Resizable ticket slide panel with drag-to-resize handle; width persisted to localStorage across page loads - Charts: submission timeline, org bar chart, and escalation breakdown - Date-range filtering propagated from the URL query string - Dark SOC-analyst CSS theme consistent with the rest of the PISCES UI (me.css)
Mount the new mantis-explorer app at /mantis-explorer in the DispatcherMiddleware and add a hub card linking to it with a fa-user-graduate icon. Also bumps locked dependency versions: rich 14→15, ruff 0.15.6→0.15.12, pre-commit 4.5.1→4.6.0, geoip2 >=4.8.0→>=5.2.0, pytest >=9.0.3.
app.py imported TICKETS_BY_ID from apps.mantis_explorer.data, but that symbol was never re-exported there, causing an ImportError on startup. Adds the import alongside the existing _raw_tickets alias.
Moving match clauses from `must` to `filter` context skips relevance scoring, which improves query performance and enables result caching at the shard level in OpenSearch. Added `?timeout=30s` to the query URL so long-running queries fail fast instead of hanging indefinitely. The filter-context change is applied consistently across `build_base_query`, `list_sensors`, and `list_log_types`.
Five tests were checking query structure under the `must` key. Now that `build_base_query` emits clauses under `filter`, the assertions are updated to match.
…n every query `load_filters()` was re-reading and re-parsing every YAML file in `filters/` on each call. A module-level dict now caches the result keyed by `(filters_dir, municipality)`. The cache is invalidated whenever the highest mtime across all YAML files changes, so filter edits are still picked up without a process restart.
…ild_base_query `module.build_extra_must()` returns a `(clauses, extra_data)` tuple. `--dump-query` was passing the full tuple as `extra_must`, so the generated query body contained a tuple instead of a list of ES clauses. Destructure with `extra_must, _` to extract only the clauses.
…on handler The bare `except Exception` in `run_cross_protocol_query` was silently swallowing per-protocol failures, making it impossible to diagnose which protocol was erroring and why. Now logs the failing protocol name and exception message via the Rich console.
get_tickets_for_ip() was iterating over all raw tickets on every request to filter by IP membership. It now uses the pre-built TICKETS_BY_IP and TICKETS_BY_ID dicts for O(1) lookup. Startup row lists (MALICIOUS_ROWS, FP_ROWS, INFRA_ROWS, DNS_RESOLVER_ROWS, UNDETERMINED_ROWS) are now pre-sorted at module load time by each table's default sort key, so repeated calls hit Timsort's O(n) already-sorted fast path instead of a full O(n log n) sort on every request.
mantis_index.py: wrap all API calls in a requests.Session for connection pool reuse. When total_count is known upfront, remaining pages are now fetched concurrently via ThreadPoolExecutor(max_workers=8) and assembled in page order for a stable index. Falls back to sequential pagination only when total is unknown and no page cap is set. mantis_search.py: switch search_via_api() to requests.Session so all pages within a single search share one TCP connection. Also removes student_activity.py — a 795-line near-identical duplicate of activity_report.py with no external callers.
Four cases: normal lookup returns tickets sorted newest-first, empty IP returns an empty list, missing ticket IDs in TICKETS_BY_ID are silently skipped, and results are stable when multiple tickets share a created_at.
Add a new Tickets tab backed by a tickets Blueprint that surfaces escalation rates, institution breakdowns, and ticket volume timeline from mantis_explorer data. Add a sensor browser modal (satellite-dish button) that fetches hedgehog sensor activity via a new /api/dashboard/sensors endpoint, renders a proportional bar list, and threads the selected sensor filter through the OpenSearch and overview sections. A badge on the toolbar button shows how many sensors are active.
Register the tickets Blueprint in app.py and add the /api/dashboard/sensors route that serves sensor_summary.html content. In dashboard.html: add the Tickets tab button, sensor-badge button with JS state (_selectedSensors, getSensorParam), date picker row that appears for Mantis and Tickets tabs, and openSensorModal / applySensorSelection helpers. In base.html: add sensor-modal markup (backdrop + panel), CSS for .sensor-badge and #sensor-btn, pill-style sub-tab bar, uniform design tokens (--radius-*, height:28px inputs, rounded toolbar), and dark/light color-scheme for native date inputs.
…a charts Remove agg_opensearch_protocols (single-window terms agg on event.dataset) and replace with three date_histogram aggregations: agg_notice_over_time, agg_suricata_over_time, and agg_conn_volume_over_time. All three accept an optional sensors filter and select an appropriate interval via _interval_for_range. Add agg_suricata_alert_count (excluding SURICATA STREAM* noise), agg_new_ips_delta (current vs previous window unique public IPs), and parse_sensors (comma-separated sensor param → list|None). Thread the sensor param through the opensearch section route and include it in the cache key.
…nify to hbars Remove panels that added noise without actionable insight: DNS query types and rcodes, HTTP methods, SSL versions and cipher suites, and conn_states. Remove the donut chart renderer entirely — all remaining charts use the horizontal bar (hbar) helper for visual consistency. In section.html, replace the protocol breakdown bar chart with three stacked area charts (notices, Suricata alerts, connection volume) using the new over-time data from the aggregation layer.
…workqueues Replace the verdict donut + summary table with an alert trend area chart (Zeek Notices and Suricata Alerts on one axis) and two new actionable tables: "IPs Without Tickets" and "Untriaged High-Activity IPs", both derived from the cross-protocol query. agg_cross_source_ips now returns untriaged, no_ticket, alerts_no_ticket, and total_ips alongside the main opensearch list, and exposes per-IP notice and suricata hit counts (from per_protocol in run_cross_protocol_query). agg_overview fetches notice/suricata over-time and agg_new_ips_delta in parallel and propagates the sensor filter to all concurrent tasks. The section route and cache key are updated to include the sensor param.
…ts panel Replace the time_range query param with since/until date pickers so analysts can filter by an explicit calendar window rather than a rolling OpenSearch interval (Mantis data is indexed by created_at, not @timestamp). Add _filter_tickets and _filter_malicious helpers that apply since/until bounds to _raw_tickets and MALICIOUS_ROWS respectively. All public aggregation functions (agg_mantis_attack_types, agg_mantis_timeline, agg_mantis_top_ips) now accept and apply these filters. Remove agg_mantis_blocklists — the blocklist sources chart was noisy and rarely actionable; its space is given to an enlarged timeline chart.
Replace stale Mantis and Kibana screenshots with current Dashboard and Threat Model images. Update the app table to list all four apps — OpenSearch, Threat Model, Dashboard, and Mantis Explorer — with accurate descriptions matching the renamed panels.
The app's scope has always been broader than Mantis ticket browsing — it covers threat modelling, IP verdict classification, FP/TP tracking, and blocklist analysis. The old name caused confusion and leaked an implementation detail (Mantis) into the app identity. - Rename apps/mantis_web/ → apps/threat_model/ (all files) - Rename mantis_web_run.py → threat_model_run.py and update shim path - Change dispatcher mount from /mantis to /threat-model in run_all.py - Update hub card link and all cross-app imports to apps.threat_model - Rename tests/test_mantis_web_helpers.py → test_threat_model_helpers.py - Update docs/advanced-usage.md standalone launcher command
…ness Replace the card grid with a compact list layout for better scanability. Add a timestamp row showing how recently the ticket index and threat model were updated, read directly from local data files — no API calls required.
The per-app shims (opensearch_web_run.py, threat_model_run.py, dashboard_web_run.py) are superseded by run_all.py. Remove them and drop the standalone launch examples from advanced-usage.md.
Remove the --retrain --classify-stats --use-ml flags from the command snippet. The plain invocation is the correct default and the flags were stale guidance.
Reads the project version from pyproject.toml at startup and fetches origin to compute how many commits the running instance is behind its remote branch. The footer on the landing page now displays the version and either an up-to-date indicator or a commits-behind warning, giving operators visibility into whether the deployed instance is current.
Show a notice modal on page load warning that the escalated ticket count is not yet tuned enough for real reporting. Includes first-run command hint (uv run src/mantis/mantis_index.py). Matches the existing threat model notice modal pattern.
…heme, and logos Introduces make_shared_static_blueprint() which mounts apps/shared/static/ at /shared/static/ across all Flask apps. The directory contains: - tokens.css — MD3 design tokens and CSS custom properties - base.css — reset, layout, component base styles - theme.js — theme toggle logic and flash-prevention inline script - pisces-logo.ico / pisces-logo.png — canonical logo assets Having a single source for these assets means token/component changes propagate everywhere without touching individual app stylesheets.
…cate files
All five apps (hub, opensearch_web, mantis_explorer, threat_model,
dashboard_web) now register make_shared_static_blueprint() so they
serve tokens.css, base.css, theme.js, and logos from /shared/static/.
Per-app changes:
- app.py: register shared_static blueprint
- templates: favicon, logo, and theme script tags point to /shared/static/;
inline theme-flash-prevention IIFE and toggleTheme() removed (now in
theme.js)
- CSS: tokens, CSS custom properties, reset, and base component styles
stripped — each file now contains only app-specific overrides
Ten duplicate logo files deleted (~870 KB saved):
apps/{hub,opensearch_web,mantis_explorer,threat_model,dashboard_web}/static/pisces-logo.{ico,png}
Remove the dark/light theme toggle button from the nav bar and sidebar in all five web apps (OpenSearch, Threat Model, Mantis Explorer, Dashboard, Hub). Theme selection is moving to a centralised settings page in the Hub. Also removes the now-unused .btn-theme CSS block from base.css.
Add a /settings route to the Hub with a dropdown to select dark or light theme. The choice is persisted in localStorage and applies across all PISCES apps. The Hub nav bar now shows a gear icon linking to the settings page instead of the inline theme toggle button.
All dashboard ECharts (tickets, mantis, opensearch, overview) had hardcoded dark-theme hex colors for axis labels, tooltips, grid lines, and legends. On light theme these were illegible — dark text on a light background. Replace all hardcoded values with getComputedStyle reads of the design tokens (--on-surface, --on-surface-dim, --outline, --surface-container, --surface) so charts adapt to whichever theme is active.
Rename existing themes to PISCES Dark and PISCES Light, then add: - Gruvbox Dark & Light (morhetz/gruvbox palette) - Tokyo Night Dark & Light (enkia/tokyo-night-vscode-theme palette) - Catppuccin Latte, Frappé, Macchiato, Mocha (catppuccin/catppuccin) All 10 themes define the full set of design tokens in tokens.css. The settings dropdown groups themes by family using optgroups. theme.js auto-migrates users who had the old 'dark'/'light' values in localStorage to the new 'pisces-dark'/'pisces-light' names. Also updates data-theme defaults in all base templates and fixes color-scheme CSS selectors to cover all light theme variants.
The PISCES design system expects surfaces to go from darkest (surface) to lightest (surface-container-highest) for dark themes. The Catppuccin dark flavors (Frappé, Macchiato, Mocha) had Mantle mapped to surface-container-low which made it darker than the main surface, inverting the nav/sidebar contrast. Fix by shifting the hierarchy down: Crust → surface, Mantle → surface-container-low, Base → surface-container, Surface0 → surface-container-high, Surface1 → surface-container-highest. Also fix Catppuccin Latte and both Gruvbox themes which had duplicate values for surface-container-low and surface-container, giving them no visual distinction between nav bars and content backgrounds.
Use sessionStorage to track dismissal so the notice dialogue appears on first visit from the hub but not on every page navigation between overview and org views.
Adds an orange-outlined NOTICE button with a warning icon in the nav bar that re-opens the notice dialogue after it has been dismissed.
…sed controls Restructure the global search bar into a two-row layout: filter inputs on top spread across the full width, action buttons below. - Wrap each filter group in pill containers with subtle borders - Add Font Awesome icons to all filter labels - Tint Src IP and Dst IP pills with muted red/blue accents - Make share button a clickable pill matching the overall style - Remove timestamp compact/full toggle — always show full timestamps - Fix unclosed .check-label CSS rule that broke .org-icon styles - Remove unused .sep divider elements and CSS
…tton Replace the sensor text input and browse button with a single clickable pill that displays the current sensor selection and opens the modal on click. Selecting all sensors now correctly defaults to 'all' instead of listing every sensor in the URL. Simplify modal controls from 'all/none' to a single 'clear' link. Normalize all search bar pills to a consistent 32px height.
…tton Replace the standalone home icon button with a brand link that points to the hub when running under a script prefix. Applies to dashboard, mantis explorer, and threat model base templates.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Validate since/until parameters with date.fromisoformat() at the route level before they can reach exception messages rendered in templates. Previously, a crafted since/until value (e.g. <script>alert(1)</script>) would raise a ValueError whose str(exc) was passed to the template via data['error']. While Jinja2 autoescape mitigates this, defence in depth requires rejecting invalid input at the boundary. Add safe_date_param() to apps.dashboard_web and apply it in both the tickets and mantis dashboard routes. Invalid values are silently replaced with empty strings (no date filter applied).
…format Return date.fromisoformat(v).isoformat() instead of the original user string. This produces a new string from the parsed date object, which CodeQL can verify is not tainted by user input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes dev to main with 44 commits across 86 files.
Summary
This release delivers a full UI overhaul, shared theming infrastructure, performance improvements to the query and filter pipeline, and a new student activity explorer app.
Core / Performance
New: Mantis Explorer app
Threat Model rename
apps/mantis_web→apps/threat_model— directory, imports, and all references updatedDashboard redesign
OpenSearch Web
Shared static assets and theming
apps/shared/blueprint serving design tokens, base CSS, theme JS, and logosHub
CI / Quality
Tests
All 454 tests pass, ruff check clean.