Feat/llm vision#7
Merged
Merged
Conversation
…DTOs A property or meter-group name containing quotes could destabilize the chatbot's tool-call generation (a malformed name broke Groq's function-call parsing and surfaced as a 502). Names are also echoed verbatim into chatbot tool-result context, so a semantically injection-like name (no unsafe characters required) could poison the model's context. Add isSafeName and isSafeAgainstPromptInjection validators, the latter reusing chatbot.guard's phrase list as a single source of truth. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Manual red-teaming surfaced three real leaks: (1) the model would enumerate
its internal tool functions when asked conversationally ("what tools do you
have"), (2) it would agree to draft emails/documents from the user's own
data, treating topic relevance as sufficient scope even though drafting is a
different task type than answering, and (3) zero-record date ranges were
reported as a bare "0" instead of "no data found," which reads as a
confirmed reading rather than an absence of one.
- chatbot.guard.ts: add DISCLOSURE_PATTERNS as a code-level backstop (tool
names, "my instructions say...", leaked prompt XML tags) alongside the
existing jailbreak-phrase guard, since disclosure protection was
previously prompt-only with no enforcement.
- chatbot.service.ts: system prompt now explicitly scopes by task type, not
just topic (refuse drafting/composing regardless of subject), forbids
naming/describing internal tools even when asked conversationally, and
distinguishes "no data recorded" from a genuine zero reading.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The generic apiRateLimiter (1000/hr, IP-keyed) isn't cost-aware: each /chatbot call can run up to MAX_TOOL_CALL_ROUNDS LLM round-trips against the tenant's own provider key, so a shared office IP could exhaust the whole budget on one account's chatbot usage. Add chatbotRateLimiter (30/hr, keyed by authenticated userId via AuthenticatedRequest, falling back to ipKeyGenerator for the IPv6-safe IP case) ahead of the existing DTO validation on the /chatbot route. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Groq and Ollama Cloud's OpenAI-compatible endpoints diverge on how they accept image content, so LlmChatMessage.content now accepts a provider-agnostic LlmContentPart[] that LlmClient maps to the correct wire shape per provider, instead of callers building raw JSON. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Deletes gemini.lib.ts entirely — OCR (meter readings, bill photos) now routes through the user's configured vision LLM (Groq or Ollama Cloud) via the new vision-ocr.lib.ts, reusing LlmClient. No fallback: OCR endpoints 404 until a vision provider is configured. SSRF-guarded image fetching and prompt/response parsing were extracted verbatim into image-fetch.util.ts and ocr-parsing.util.ts so behavior is unchanged. All three OCR call sites (image-extraction, reading, bills, billing-cycle controllers) now thread userId through to resolve the caller's vision config. Also fixes billing-cycle's OCR endpoint returning a different raw_amount than the other two OCR endpoints — it was recomputing rate * consumption instead of passing through the OCR-extracted total, which is wrong whenever a bill has taxes or fixed charges. Removes the now-unused @google/generative-ai dependency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not every provider has a usable free vision model (e.g. Ollama Cloud), so
users need to mix providers — chat on one, vision (OCR) on another. Adds
vision_provider/vision_model/encrypted_vision_api_key fields alongside the
existing chat fields, plus PATCH /llm-config/vision to manage them.
When the vision provider matches the stored chat provider, apiKey is
optional and the chat API key is reused; when it differs, apiKey is
required (first setup, or switching to yet another distinct provider).
Switching the vision provider back to match chat clears any previously
stored distinct vision key via FieldValue.delete() — plain `undefined`
would silently no-op given this project's Firestore
ignoreUndefinedProperties setting.
GET /llm-config now returns both configs:
{provider, model, hasKey, visionProvider, visionModel, visionHasKey}.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LLM Provider settings page now has a Chatbot tab and a Vision (OCR) tab, each with its own provider/model/API key, matching the backend's independent chat + vision llm-config. The vision tab's API key field becomes optional (with a hint that it reuses the chatbot's key) whenever its selected provider matches the chatbot tab's, and required otherwise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removes remaining Gemini/GEMINI_API_KEY references and documents the new PATCH /llm-config/vision endpoint, the independent chat/vision provider model, and the UI's two-tab settings page across root CLAUDE.md, api/functions/CLAUDE.md, ui/CLAUDE.md, and API_SETUP.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tence
New photo-settings feature (one doc per user, same pattern as llm-config):
GET/PATCH /photo-settings returns/sets {savePhotos: boolean}, defaulting
to false when unconfigured (unlike llm-config, GET never 404s here).
This is enforced client-side, not by the backend — POST /readings and
/readings/batch still accept an optional image_url unchanged. The setting
exists so web and mobile can decide whether to attach a captured photo's
image_url on create, or discard it after using it for OCR suggest.
Billing-cycle/bill photos have no persistence path regardless, since
BillingCycle has no image_url field to begin with.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Capturing a meter photo now auto-calls POST /readings/ocr to suggest the reading amount — previously the capture flow had no OCR at all, and embedded the raw base64 photo directly into image_url on every submit. image_url is now only attached to the create/batch payload when the new savePhotos preference (GET /photo-settings) is enabled — off by default, so the photo is used in-memory for the OCR call and then stripped right before submission. Settings screen gained a toggle for this preference. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eadings New PhotoDropzone component (click or drag to add/replace a photo) replaces the batch tab's separate Upload Photo + Suggest buttons and the manual tab's plain image-URL text field — both tabs now use the same control. Selecting or dropping a photo auto-runs OCR suggest immediately; there's no standalone Suggest button anymore. image_url is now only attached to the create payload (batch and manual) when the new savePhotos preference is enabled, loaded once on mount via GET /photo-settings. Adds /settings/photos with a toggle for it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the Photo Settings API entry and updates the Mobile Screens summary for CaptureReadings' new auto-OCR-suggest behavior and the Settings save-photos toggle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Caps fetched image size and allowlists MIME types (unbounded consumption), validates OCR-extracted date fields and wraps JSON.parse safely (improper output handling), adds a system prompt plus a chatbot.guard.ts-style regex check to resist prompt injection from image content, and confirms/cleans up the Ollama Cloud multimodal payload shape. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Documents what data Utilitool collects, that OCR photos are sent to the user's own configured Groq/Ollama vision provider, those providers' stated retention/training policies, and the single-tenant (shared-org) data model so it's clear no per-landlord isolation exists by design. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a second, irreversible DELETE /:id/purge step (admin-only) on top of the existing soft-delete, so right-to-erasure requests can be fulfilled without leaving orphaned child records. Cascades purge to already-archived children the same way the existing archive cascade works. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Server-side 403s non-admin requests to POST /chatbot; the web ChatWidget is hidden for non-admin users to match, since mobile has no chatbot UI. Tool-call results also tokenize Property.room_name before it reaches the LLM provider, unmasking only in the final reply shown to the user, to minimize business data sent to third-party providers. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-encodes fetched images via sharp before they're sent to the third-party vision provider, dropping EXIF data (including GPS location tags) while preserving correct visual orientation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates PRIVACY.md to reflect the property-name masking, EXIF stripping, and admin-only chatbot restriction now in place, plus a recommendation to enable Groq Zero Data Retention. Updates api/CLAUDE.md and ui/CLAUDE.md with matching architecture notes for the purge lifecycle and chatbot gating. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…l-regex Disable linebreak-style since core.autocrlf=true causes Windows checkouts to conflict with the google config's LF requirement, and suppress the intentional no-control-regex flag in sanitize.util.ts's unsafe-name check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI's lint step exits nonzero on Prettier warnings (unlike the local run), so these files needed reformatting to keep the pipeline green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.