Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ flowchart LR
- `logsight.logbert.LogBertAnomalyAdapter` — bounded local-checkpoint masked-LM adapter with calibrated loss-threshold decisions.
- `logsight.onnx_worker.OnnxInferenceWorker` — bounded ONNX Runtime feature-batch worker with model provenance and score validation.
- `logsight.redis_control.RedisAdmissionController` — atomic Redis deduplication and per-service rate-limit admission control.
- `logsight.distributed.CeleryInferenceDispatcher` — bounded JSON Celery dispatch for asynchronous embedding work.
- `logsight.streaming.InMemoryStream` — bounded local async stream for tests and development.
- `logsight.streaming.KafkaStream` — optional Kafka/Redpanda producer using `aiokafka`.
- `logsight.storage.ClickHouseStore` and `QdrantStore` — HTTP adapters for durable analytical and semantic storage.
Expand All @@ -80,6 +81,7 @@ Services exposed locally:
| ClickHouse | 8123 | Log analytics/storage |
| Qdrant | 6333 | Vector similarity search |
| Redis | 6379 | Cache/dedup/rate limiting foundation |
| Celery worker | — | Bounded asynchronous embedding inference |
| OTel Collector | 4317 / 4318 | OTLP trace and log intake |
| Prometheus | 9090 | Metrics collection |
| Grafana | 3000 | Dashboards |
Expand All @@ -92,6 +94,23 @@ pip install -e ".[api,kafka,observability,ai,parsing,redis]"

The optional integrations are deliberately separated so a developer can use the original local-first CLI without downloading heavyweight ML or infrastructure clients.

### Distributed inference workers

The enterprise Compose stack includes an idle Celery embedding worker. Its task
contract accepts bounded JSON envelopes only, limits prefetching to one task, and
uses late acknowledgement/reject-on-worker-loss semantics. API dispatch is
disabled until a deployment provisions an approved model artifact:

```bash
export LOGSIGHT_CELERY_DISPATCH_ENABLED=true
export LOGSIGHT_EMBEDDING_MODEL=/models/approved-embedding-model
```

After enabling it, accepted API events include an `inference_task_id`; asynchronous
embedding does not change the request-time anomaly score. See
[Celery worker deployment](docs/CELERY_WORKERS.md) for queue, security, and
production-promotion controls.

### OpenTelemetry Collector and trace–log correlation

The enterprise reference stack includes an OTLP Collector. Set
Expand Down Expand Up @@ -218,7 +237,7 @@ Pull requests retain the existing formatting, linting, strict typing, unit/integ
- [x] ONNX Runtime inference worker
- [x] Redis-backed deduplication/rate limiting
- [x] OTel Collector deployment and trace-log correlation pipeline
- [ ] Celery/Ray distributed inference workers
- [x] Celery/Ray distributed inference workers
- [ ] Helm chart with HPA/PDB/network policies
- [ ] RAG incident investigator with versioned runbooks/commits
- [ ] Labeled benchmark corpus with precision/recall and alert-burden metrics
Expand All @@ -228,6 +247,7 @@ Pull requests retain the existing formatting, linting, strict typing, unit/integ
- [Production audit](docs/AUDIT.md)
- [Architecture](docs/architecture.md)
- [Collector correlation](docs/OTEL_COLLECTOR.md)
- [Celery inference workers](docs/CELERY_WORKERS.md)
- [Deployment and rollback checklist](docs/DEPLOYMENT.md)
- [Benchmark methodology](docs/BENCHMARKING.md)
- [Mathematical foundations](docs/MATHEMATICAL_FOUNDATIONS.md)
Expand Down
28 changes: 27 additions & 1 deletion docker-compose.enterprise.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ services:
build:
context: .
args:
INSTALL_EXTRAS: api,redis,observability
INSTALL_EXTRAS: api,redis,observability,workers
image: logsight-ai:enterprise
entrypoint: []
command: ["uvicorn", "logsight.api:app", "--host", "0.0.0.0", "--port", "8000"]
Expand All @@ -13,6 +13,9 @@ services:
LOGSIGHT_RATE_LIMIT: "120"
LOGSIGHT_RATE_WINDOW_SECONDS: "60"
LOGSIGHT_DEDUP_TTL_SECONDS: "300"
LOGSIGHT_CELERY_BROKER_URL: redis://redis:6379/0
LOGSIGHT_CELERY_RESULT_BACKEND: redis://redis:6379/1
LOGSIGHT_CELERY_DISPATCH_ENABLED: "false"
LOGSIGHT_OTEL_ENABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
OTEL_SERVICE_NAME: logsight-api
Expand All @@ -34,6 +37,29 @@ services:
tmpfs:
- /tmp:size=32m,mode=1777

logsight-worker:
build:
context: .
args:
INSTALL_EXTRAS: workers,ai
image: logsight-ai:inference-worker
entrypoint: []
command: ["celery", "-A", "logsight.distributed.celery_app", "worker", "--loglevel=INFO", "--concurrency=2", "--prefetch-multiplier=1", "--queues=inference"]
environment:
LOGSIGHT_CELERY_ENABLED: "true"
LOGSIGHT_CELERY_BROKER_URL: redis://redis:6379/0
LOGSIGHT_CELERY_RESULT_BACKEND: redis://redis:6379/1
LOGSIGHT_CELERY_MAX_BATCH_SIZE: "64"
LOGSIGHT_CELERY_TASK_TIME_LIMIT_SECONDS: "120"
depends_on:
- redis
read_only: true
security_opt:
- no-new-privileges:true
cap_drop: [ALL]
tmpfs:
- /tmp:size=128m,mode=1777

otel-collector:
image: otel/opentelemetry-collector-contrib:0.120.0
command: ["--config=/etc/otelcol-contrib/collector.yml"]
Expand Down
42 changes: 42 additions & 0 deletions docs/CELERY_WORKERS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Celery Distributed Inference Workers

## Contract

`logsight.distributed.CeleryInferenceDispatcher` submits bounded JSON-only
`logsight.embed_events` tasks. Each item contains the event fingerprint, message,
and optional trace/span IDs; it never sends raw log lines or arbitrary Python
objects through the broker. The worker validates task shape, batch size, and
message size before invoking the existing embedding worker.

The Celery configuration uses JSON serialization, one-task prefetching, late
acknowledgement, rejection after a lost worker, and soft/hard task limits. These
settings make retries explicit and prevent one worker from hoarding inference work.

## Reference stack

`docker-compose.enterprise.yml` defines an idle `logsight-worker` service that
uses Redis database 0 as the broker and database 1 as the result backend. It is
built with the `workers,ai` extras and starts with two worker processes.

API dispatch is deliberately disabled by default:

```bash
export LOGSIGHT_CELERY_DISPATCH_ENABLED=true
export LOGSIGHT_CELERY_BROKER_URL=redis://redis:6379/0
export LOGSIGHT_CELERY_RESULT_BACKEND=redis://redis:6379/1
export LOGSIGHT_EMBEDDING_MODEL=/models/approved-embedding-model
```

Enable dispatch only after a model artifact is provisioned and the worker image
has access to it. With dispatch enabled, accepted API events include
`inference_task_id`; inference remains asynchronous and does not alter the
request-time anomaly score.

## Production promotion

Use separate Redis users/databases, TLS, authentication, network policies, and
queue-depth/worker-health alerts. Set concurrency from measured model memory and
latency, retain task results only as long as needed, and use a deployment-owned
immutable model artifact. This implementation provides the Celery path; Ray
orchestrators can be added later behind the same bounded task envelope rather
than exposing runtime objects over the broker.
26 changes: 20 additions & 6 deletions logsight/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@

from __future__ import annotations

import asyncio
import os
from collections import deque
from dataclasses import asdict
from typing import Any

from .distributed import CeleryInferenceDispatcher, build_celery_dispatcher
from .enterprise import EnrichedLog, enrich, score_event
from .observability import configure_tracing, correlate_with_active_span, span
from .parser import parse_line
Expand Down Expand Up @@ -41,9 +43,10 @@ class BatchPayload(BaseModel): # type: ignore[misc]
_metrics = {"logsight_ingested_total": 0, "logsight_anomalies_total": 0}
_template_parser: TemplateParser = build_template_parser()
_admission_controller: RedisAdmissionController | None = build_redis_controller()
_inference_dispatcher: CeleryInferenceDispatcher | None = build_celery_dispatcher()


async def _process(item: LogPayload) -> tuple[EnrichedLog, Any]:
async def _process(item: LogPayload) -> tuple[EnrichedLog, Any, str | None]:
with span("logsight.ingest", service=item.service, source=item.source) as _:
event = correlate_with_active_span(
enrich(
Expand All @@ -64,7 +67,12 @@ async def _process(item: LogPayload) -> tuple[EnrichedLog, Any]:
_metrics["logsight_ingested_total"] += 1
if result.score >= 0.5:
_metrics["logsight_anomalies_total"] += 1
return event, result
task_id = (
await asyncio.to_thread(_inference_dispatcher.submit_embeddings, [event])
if _inference_dispatcher is not None
else None
)
return event, result, task_id


def create_app() -> Any:
Expand All @@ -85,15 +93,21 @@ async def metrics() -> Response:

@app.post("/v1/logs")
async def ingest(payload: LogPayload) -> dict[str, Any]:
event, result = await _process(payload)
return {"event": event.to_dict(), "anomaly": asdict(result)}
event, result, task_id = await _process(payload)
response: dict[str, Any] = {"event": event.to_dict(), "anomaly": asdict(result)}
if task_id is not None:
response["inference_task_id"] = task_id
return response

@app.post("/v1/logs/batch")
async def ingest_batch(payload: BatchPayload) -> dict[str, Any]:
results = []
for item in payload.logs:
event, result = await _process(item)
results.append({"event": event.to_dict(), "anomaly": asdict(result)})
event, result, task_id = await _process(item)
response: dict[str, Any] = {"event": event.to_dict(), "anomaly": asdict(result)}
if task_id is not None:
response["inference_task_id"] = task_id
results.append(response)
return {"count": len(results), "results": results}

@app.get("/v1/logs/recent")
Expand Down
Loading
Loading