Welcome to ClearShelf, a state-of-the-art retail inventory optimization and demand forecasting platform. ClearShelf blends traditional machine learning with a collaborative, multi-agent AI council powered by CrewAI to provide retailers with both mathematical precision and real-world qualitative context.
This document serves as an exhaustive, beginner-friendly guide that explains the system's inner workings, underlying architectures, workflows, current technical limitations (flaws), recent engineering improvements, and future roadmaps. Whether you are a business stakeholder, a beginner developer, or a non-technical enthusiast, this guide will walk you through the system.
In traditional retail, predicting how many units of a product (e.g., winter jackets, hiking boots, or umbrellas) will sell tomorrow is a major challenge:
- Under-stocking leads to empty shelves, disappointed customers, and lost revenue.
- Over-stocking ties up valuable capital in unsold inventory and increases storage costs.
Traditional forecasting relies on historical sales data. A mathematical model looks at the last 30 days of sales, identifies a trend (e.g., sales are going up by 2% daily), checks for weekday seasonality (e.g., weekends sell more than weekdays), and projects tomorrow's number.
However, mathematical models are blind to the real world. They don't know that:
- ☔ Tomorrow's forecast predicts a major rainstorm.
- 🔥 A viral social media campaign just launched.
- 📉 A competitor lowered their prices.
ClearShelf solves this by combining two distinct layers:
- The Mathematical Baseline: Uses machine learning (
scikit-learn's Linear Regression) to fit a rolling demand curve based on historical transactions. - The Qualitative AI Council: Coordinates a team of specialized AI agents (
CrewAI) to ingest external data streams (like weather reports and social media sentiment) and calculate a percentage adjustment to apply to the mathematical baseline.
ClearShelf is designed as a service-oriented web application consisting of a modern, glassmorphic React frontend, an asynchronous FastAPI backend, a relational database, and an LLM-powered multi-agent layer.
graph TD
A[React + TS Frontend] <-->|HTTP REST / WebSockets| B[FastAPI Backend]
B --> C[(SQLAlchemy - SQLite / Neon PostgreSQL)]
B --> D[Mathematical ML Engine]
B --> E[CrewAI Agents Council]
subgraph "Mathematical Baseline"
D --> D1[Linear Regression Model]
end
subgraph "Qualitative AI Council"
E --> E1[Data Analyst Agent]
E --> E2[Market Scout Agent]
E --> E3[Weather Analyst Agent]
E --> E4[Synthesizer Agent]
end
E2 --> F[Social Media Buzz Service]
E3 --> G[Weather Service]
- Frontend (Vite + React + TS): A premium, highly interactive dashboard that lets users manage products, track warehouse storage, monitor suppliers, upload CSV records, and trigger forecasts. It utilizes WebSockets to stream the step-by-step thinking logs of the AI agents in real-time.
- Backend (FastAPI): A high-performance Python framework that exposes RESTful endpoints for inventory management and handles the orchestration of mathematical and agentic pipelines.
- Database (SQLAlchemy + SQLite/PostgreSQL): A relational storage layer that holds information about products, historical transactions, generated forecasts, and upload history.
- Agentic Layer (CrewAI): A multi-agent framework that defines roles, backstories, and tasks for AI agents, prompting them to collaborate sequentially to reach a final forecast consensus.
The workflow of the system is divided into three distinct phases: Ingestion, Forecasting, and WebSocket Streaming.
sequenceDiagram
autonumber
actor User as Store Manager
participant FE as React Frontend
participant BE as FastAPI Backend
participant DB as Relational Database
participant ML as ML Engine (Linear Regression)
participant Crew as CrewAI Agent Council
User->>FE: Uploads Transaction CSV
FE->>BE: POST /api/upload/import (File bytes)
BE->>BE: Compute SHA-256 Hash of raw file
BE->>DB: Check if Hash already exists (Deduplication)
alt Hash is Duplicate
BE-->>FE: Return 400 Bad Request (Error Alert)
else Hash is Unique
BE->>BE: Pandas EDA: Align columns & clean nulls
BE->>DB: Bulk insert transactions & products
BE-->>FE: Return 200 OK (Import Success)
end
User->>FE: Selects product & clicks "Trigger Forecast"
FE->>BE: POST /api/forecast/trigger (Product ID, AI Enabled)
BE->>DB: Retrieve 30-day sales history
alt History < 30 days
BE->>BE: Auto-pad database with historical seasonality & random noise
end
BE->>ML: Fit Linear Regression on 30-day sales history
ML->>BE: Return Baseline ML Prediction (e.g., 50 units)
rect rgb(20, 20, 30)
Note over BE, Crew: AI Council Deliberation (If enabled)
BE->>Crew: Launch sequential CrewAI run
loop Live Log Streaming
Crew->>BE: Capture sys.stdout print statements
BE-->>FE: Stream agent reasoning logs via WebSockets (ws://)
FE->>User: Display logs in real-time terminal console
end
Crew->>BE: Return Consensus Synthesis (Markdown Report + Adjusted Qty)
end
BE->>DB: Save baseline and adjusted quantities in forecasts table
BE-->>FE: Return 200 OK (Forecast Results)
FE->>User: Update Dashboard Charts & Forecast History
When a user triggers an AI-enriched forecast, the backend coordinates a panel of four virtual specialists using CrewAI. Each agent is assigned a unique role, goal, and backstory:
- Role: Senior Database Analyst & Trend Detector.
- Goal: Analyze 30-day historical transaction records to identify statistical indicators.
- Behavior: Evaluates sales trajectories, calculates the average baseline demand, and runs a seasonality check to calculate how much sales rise during weekends (Friday through Sunday) compared to weekdays.
- Role: Brand Sentiment & Social Trend Analyst.
- Goal: Monitor current promotional campaigns and consumer sentiment.
- Behavior: Evaluates social media activity (weekly mentions, buzz scores, active promotional campaigns). If a promotion is running or sentiment is highly positive, the scout recommends a positive demand adjustment.
- Role: Meteorological Impact Assessor.
- Goal: Correlate tomorrow's weather predictions with category-specific consumer habits.
- Behavior: Ingests meteorological indicators (temperature, condition, precipitation probability). If a rainstorm is coming and the product is winter boots or jackets, it recommends a positive adjustment; if the weather is warm and dry, it reduces jacket projections.
- Role: Inventory Strategy Director.
- Goal: Reconcile mathematical baseline forecasts with qualitative agent recommendations.
- Behavior: Reviews the reports generated by the Data Analyst, Market Scout, and Weather Analyst. It calculates the final percentage adjustment, combines it with the ML baseline, and generates a formatted Markdown synthesis report detailing the rationale.
To allow developers and users to explore the platform without incurring OpenAI or Groq API costs, ClearShelf features a built-in High-Fidelity Simulation Mode.
- How it works: If the system detects that no valid
OPENAI_API_KEYorGROQ_API_KEYis configured in the.envfile, the backend falls back torun_simulated_crewinside crew.py. - Determinism: It uses the hash of the target SKU and date strings to seed the random number generators, ensuring that the simulated reports, weather states, and social media scores remain deterministic for a given product on a given day.
- Visual Fidelity: The simulator introduces artificial delays (
time.sleep) to mimic the real-time reasoning and communication lag of actual LLM processes, streaming realistic step-by-step logic logs over the WebSockets connection to the frontend.
While ClearShelf provides a robust demonstration of hybrid forecasting, a real-world enterprise deployment requires addressing several architectural and algorithmic limitations:
- The Issue: The core mathematical engine uses a standard Linear Regression model.
- Why it's a flaw: Linear Regression assumes a straight-line relationship over time. While it captures general upward/downward trajectories and simple weekday seasonality, it fails to model complex, non-linear forecasting patterns such as cyclic monthly/annual seasonality, holiday demand spikes (e.g., Black Friday), promotional decay curves, or multi-collinearity.
- Impact: Projections for highly volatile products can over- or under-estimate baseline sales.
- The Issue: The weather forecasting service (weather_service.py) and social media buzz service (social_media_service.py) return mocked data.
- Why it's a flaw: The weather and social sentiment data are generated locally using pseudo-random variables seeded by the date and SKU. The system is not connected to a live weather radar API or an actual social scraper.
- Impact: The system cannot respond to sudden, real-world unexpected weather events or genuine social media viral outbreaks unless the mock data is manually updated or replaced with actual API connectors.
- The Issue: CrewAI's task execution loop is synchronous and blocking.
- Why it's a flaw: In Python, standard LLM requests wait for the API response. In forecast.py, the
trigger_forecastendpoint handles this by executing the service in a thread pool using Starlette/FastAPI's dependency injections, which prevents blocking the main event loop. However, under high concurrent request volume (e.g., hundreds of managers querying different products simultaneously), the thread pool can saturate, leading to API latency and resource exhaustion. - Impact: Scalability is constrained; a dedicated message queue (like Celery with Redis/RabbitMQ) is needed for asynchronous task management.
- The Issue: The ML model is re-trained from scratch on every single request.
- Why it's a flaw: Instead of loading a pre-trained model file (like a serialized
.pklfile) or performing incremental training, the service queries the database, extracts the last 30 transaction rows, and fits the Scikit-Learn regressor on-the-fly. - Impact: This increases database read overhead and processing latency for each forecasting request.
- The Issue: The system tracks inventory as a unified stock quantity.
- Why it's a flaw: Real-world retailers operate multi-branch networks with separate warehouses, regional distribution centers, and physical storefronts. ClearShelf stores a single
current_stockvalue in theproductstable and does not support regional transfer logic or multi-location demand distribution. - Impact: Inventory cannot be optimized across multiple geographical nodes.
During development, several key improvements were implemented to enhance stability, user experience, and data safety:
- SHA-256 Cryptographic File Deduplication:
To prevent duplicate transactional uploads (which would corrupt the rolling history and bias the ML models), the backend computes the SHA-256 hash of the uploaded CSV bytes in upload.py. If the hash is found in
upload_history, the backend terminates the write and alerts the user, ensuring data integrity. - Thread-Safe WebSocket Log Interception:
To capture the standard output from CrewAI and stream it to the frontend over WebSockets, the system implements a custom
WebSocketStreamwrapper that overridessys.stdoutduring active runs. It callsmain_loop.call_soon_threadsafe(...)in forecast_service.py to broadcast logs safely across thread boundaries back to the main event loop. - Automatic 30-Day Database Seeding:
Forecasting models require a baseline history. If a user creates a new product that lacks transactional data, the database's
ensure_30_days_historyservice automatically runs back in time to seed 30 days of mock sales, complete with weekend multipliers and uniform random noise, ensuring the system can forecast immediately. - Smart Column Schema Alignment:
The CSV ingestion engine is equipped with flexible naming checks. Using Pandas, it maps variations of column headers (such as
mrp,unit_price, orpricemapping toprice) automatically, preventing import failures due to simple formatting mismatches.
To transition ClearShelf from a high-fidelity prototype to an enterprise-grade SaaS platform, the following upgrades are planned:
Replace standard Linear Regression with state-of-the-art forecasting models:
- Prophet (by Meta): Excellent for handling daily, weekly, and yearly seasonalities, holiday effects, and structural trend shifts.
- DeepAR / LSTM (Long Short-Term Memory): Neural networks designed to capture sequential dependencies and non-linear patterns over time.
Connect services to live external REST APIs:
- OpenWeatherMap API: Pull live 5-day weather forecasts for the store's physical zip code.
- Social Scrapers: Query Reddit APIs and Twitter/X keyword tracking endpoints to calculate genuine sentiment indices.
- E-Commerce Sync: Synchronize stock counts directly with Shopify or WooCommerce webhooks.
Bridge forecasting with supplier logic:
- When the system calculates that
current_stockwill fall below thetotal_7day_forecastquantity, it automatically generates a pending Purchase Order (PO) in the database. - The PO is calculated based on the supplier's average lead time and automatically dispatched to the supplier's contact email.
- Introduce Celery with Redis to offload forecasting jobs.
- When a user clicks "Trigger Forecast", FastAPI will push the job to Redis and return a task ID. The dashboard will monitor the progress of the worker task asynchronously, eliminating the risk of thread-blocking.
Follow these steps to run ClearShelf on your local machine:
- Python 3.10+
- Node.js v18+
- PostgreSQL (Optional; local SQLite is used by default if no connection string is provided)
- Navigate into the
backend/directory:cd backend - Create and activate a virtual environment:
- Windows:
python -m venv venv .\venv\Scripts\activate - macOS / Linux:
python3 -m venv venv source venv/bin/activate
- Windows:
- Install dependencies:
pip install -r requirements.txt
- Configure environment variables in a
.envfile:DATABASE_URL=your-optional-postgresql-url OPENAI_API_KEY=your-openai-api-key # Optional: Leave blank to use simulator mode OPENAI_MODEL_NAME=gpt-4o GROQ_API_KEY=your-groq-api-key # Optional: Alternative LLM GROQ_MODEL_NAME=llama-3.1-70b-versatile
- Start the backend development server:
The database will be automatically initialized and seeded with mock inventory and sales records. The API documentation will be available at
uvicorn app.main:app --reload
http://localhost:8000/docs.
- Navigate into the
frontend/directory:cd ../frontend - Install the required Node packages:
npm install
- Start the Vite development server:
The web application will launch at
npm run dev
http://localhost:5173.
You can start both systems quickly using the root scripts:
- Double-click
run_backend.bator execute.\run_backend.ps1in PowerShell to launch the API server. - Navigate to the frontend and run
npm run devto start the interface.