Skip to content

JYJiahaoYan/InstrumentReservationSystem

BranchesTags

Repository files navigation

Experimental Instrument Reservation System

A production-ready microservices system built with Spring Boot and Spring Cloud for managing time-based reservations of shared scientific instruments (SEM/TEM, AFM, XRD, cryostats, lasers, cleanroom tools, GPU nodes, etc.) in research laboratories.

🎯 Features

  • Time-based Reservations with conflict detection (no double-booking)
  • Approval Workflow (instrument owner approval required)
  • Training Gates (required certification before booking)
  • Session Tracking (check-in/check-out) and usage-based billing
  • Event-driven Notifications via Apache Kafka
  • Comprehensive Observability with Prometheus, Grafana, Loki, and Zipkin
  • Data Engineering & Analytics with ETL pipelines, data warehouse, and business intelligence
  • OAuth2 Security with Keycloak integration
  • Service Discovery with Netflix Eureka
  • Centralized Configuration with Spring Cloud Config

🏗️ Architecture

The system follows a microservices architecture pattern with the following components:

┌─────────────┐
│   Gateway   │ (Spring Cloud Gateway)
└──────┬──────┘
       │
       ├───► User Service (MongoDB)
       ├───► Product/Instrument Service (PostgreSQL)
       ├───► Order/Reservation Service (PostgreSQL)
       ├───► Notification Service
       └───► Analytics API (PostgreSQL Analytics DB)
       
Infrastructure:
- Config Server (Spring Cloud Config)
- Eureka Server (Service Discovery)
- Keycloak (OAuth2/OIDC)
- Kafka (Event Streaming)
- Kafka Connect (Event to Data Lake)
- PostgreSQL (OLTP + OLAP Analytics DB)
- MongoDB (User Data)
- MinIO (Data Lake - S3-compatible)
- Prometheus + Grafana (Metrics & Analytics Dashboards)
- Loki (Logs)
- Zipkin (Distributed Tracing)

Data Pipeline:
Kafka Events → Kafka Connect → MinIO (Data Lake)
PostgreSQL/MongoDB → ETL Service → Analytics DB (Star Schema)
Analytics DB → Analytics API → Grafana Dashboards

📦 Services

Core Services

  1. Gateway Service (gateway)

    • API Gateway with routing, security, and CORS
    • Port: 8080
    • Routes requests to backend services
    • JWT token validation

  2. User Service (user)

    • User management, lab groups, roles, and training records
    • Port: 8082
    • Database: MongoDB
    • Manages researchers, PIs, instrument owners, and admins

  3. Product/Instrument Service (product)

    • Instrument catalog with policies and availability
    • Port: 8081
    • Database: PostgreSQL
    • Manages instruments, maintenance windows, and rate plans

  4. Order/Reservation Service (order)

    • Reservation lifecycle with conflict detection and approvals
    • Port: 8083
    • Database: PostgreSQL
    • Handles reservations, sessions, and approval workflows

  5. Notification Service (notification)

    • Event-driven notifications via Kafka
    • Port: 8084
    • Sends email notifications for reservation events

  6. Analytics API (analytics-api)

    • Business intelligence and analytics queries
    • Port: 8085
    • Database: PostgreSQL (analyticsdb)
    • Provides pre-built analytics endpoints and custom SQL query execution
    • Role-based access: ADMIN, ANALYST

  7. Analytics Service (analytics-service)

    • ETL pipeline for data transformation
    • Extracts data from PostgreSQL, MongoDB, and MinIO data lake
    • Transforms and loads data into analytics warehouse (star schema)
    • Runs scheduled data quality checks
    • Technology: Python with pandas, SQLAlchemy

Infrastructure Services

  1. Config Server (configserver)

    • Centralized configuration management
    • Port: 8888
    • Serves configuration from Git repository

  2. Eureka Server (eureka)

    • Service discovery and registration
    • Port: 8761
    • Service registry dashboard

  3. Kafka Connect (kafka-connect)

    • Connects Kafka topics to data lake (MinIO)
    • Port: 8083
    • S3 sink connectors for event persistence

  4. MinIO (minio)

    • S3-compatible object storage for data lake
    • Ports: 9000 (API), 9001 (Console)
    • Stores raw events, metrics, logs, and database snapshots

⚙️ Configuration

Environment Variables

Services use Spring Cloud Config for centralized configuration. Local overrides can be set via:

  • application-local.yml (not committed to Git)
  • Environment variables
  • Docker environment variables

Keycloak Setup

See KEYCLOAK_SETUP.md for detailed Keycloak configuration:

  • Create realm: labreserve
  • Create client: labreserve-api
  • Configure roles: RESEARCHER, PI, INSTRUMENT_OWNER, ADMIN

Database Setup

PostgreSQL databases are automatically created via deploy/docker/init-multi-db.sql:

  • orderdb - Reservation service (OLTP)
  • productdb - Instrument service (OLTP)
  • analyticsdb - Analytics data warehouse (OLAP) with star schema

MongoDB is used for user service (database: userdb).

The analytics database uses a star schema design with:

  • Dimension tables: dim_time, dim_instruments, dim_users, dim_lab_groups, dim_experiment_types
  • Fact tables: fact_reservations, fact_sessions, fact_approvals, fact_events
  • Pre-populated time dimension (2020-2025) for time-series analysis

📡 API Documentation

Gateway Endpoints

All API requests go through the Gateway at http://localhost:8080:

User Service

  • GET /api/users/{id} - Get user details
  • POST /api/users - Create user
  • GET /api/users/{id}/training - Get training records

Instrument Service

  • GET /api/products - List instruments
  • GET /api/products/{id} - Get instrument details
  • GET /api/products/{id}/availability - Check availability
  • POST /api/products - Register instrument

Reservation Service

  • POST /api/orders - Create reservation
  • GET /api/orders/{id} - Get reservation details
  • PUT /api/orders/{id}/approve - Approve reservation
  • PUT /api/orders/{id}/reject - Reject reservation
  • POST /api/orders/{id}/check-in - Start session
  • POST /api/orders/{id}/check-out - End session

Analytics Service

  • POST /api/analytics/query - Execute custom SQL query
  • GET /api/analytics/reservations/summary - Reservation summary statistics
  • GET /api/analytics/instruments/utilization - Instrument utilization metrics
  • GET /api/analytics/instruments/latency - Instrument approval latency analysis
  • GET /api/analytics/users/activity - User activity metrics
  • GET /api/analytics/users/no-show - Users with high no-show rates
  • GET /api/analytics/reservations/failure-patterns - Patterns leading to reservation failures
  • GET /api/analytics/sessions/utilization-by-hour - Utilization by time of day
  • GET /api/analytics/reservations/peak-demand - Peak demand periods
  • GET /api/analytics/export?query=...&format=csv|json - Export query results

Authentication

All endpoints require OAuth2 JWT tokens from Keycloak. Include the token in the Authorization header:

curl -H "Authorization: Bearer <token>" http://localhost:8080/api/users/1

📊 Observability

Metrics (Prometheus)

Custom business metrics:

  • reservations.created - Reservation creation counter
  • reservations.approved - Approval counter
  • sessions.duration - Session duration histogram
  • instruments.active.count - Active instruments gauge

Dashboards (Grafana)

  • Grafana UI: http://localhost:3000
  • Default Login: admin/admin
  • Pre-configured Dashboards:
    • Reservation metrics dashboard (operational metrics)
    • Analytics Reservation Dashboard (business intelligence)
    • Analytics Instrument Utilization Dashboard
    • Analytics User Behavior Dashboard

See OBSERVABILITY_GUIDE.md for detailed metrics and dashboards.

Logs (Loki)

Example query:

{service="reservation-service"} |= "ERROR"

Tracing (Zipkin)

📈 Data Engineering & Analytics

Data pipeline for business intelligence and analytics.

Data Pipeline

Kafka Events → Kafka Connect → MinIO (Parquet, partitioned by date)
PostgreSQL/MongoDB → ETL Service → Analytics DB (Star Schema)

MinIO Data Lake: S3-compatible storage (http://localhost:9001)
Buckets: raw-events, raw-metrics, raw-logs, raw-db-snapshots

ETL Pipeline: Extracts from PostgreSQL, MongoDB, MinIO → transforms through staging → loads into analytics warehouse. Runs daily at 2 AM with data quality checks.

Analytics Warehouse: Star schema with dimension tables (SCD Type 2) and fact tables. Pre-built views for common queries.

Analytics API: REST API with pre-built endpoints and custom SQL query execution. Data export in CSV/JSON. Requires ADMIN or ANALYST role.

Example Analytics Queries

# Get reservation summary
curl -H "Authorization: Bearer <token>" \
  "http://localhost:8080/api/analytics/reservations/summary?startDate=2024-01-01&endDate=2024-12-31"

# Get instrument utilization
curl -H "Authorization: Bearer <token>" \
  "http://localhost:8080/api/analytics/instruments/utilization"

# Execute custom query
curl -X POST -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"query": "SELECT instrument_name, COUNT(*) FROM fact_reservations fr JOIN dim_instruments di ON fr.instrument_key = di.instrument_key GROUP BY instrument_name"}' \
  "http://localhost:8080/api/analytics/query"

For detailed documentation, see:

🛠️ Development

Project Structure

.
├── configserver/          # Configuration server
├── eureka/                # Service discovery
├── gateway/               # API Gateway
├── user/                  # User service
├── product/               # Instrument service
├── order/                 # Reservation service
├── notification/          # Notification service
├── analytics-api/         # Analytics API service
├── analytics-service/     # ETL pipeline service
├── deploy/
│   └── docker/           # Docker Compose and deployment configs
│       ├── docker-compose.yml
│       ├── prometheus/
│       ├── grafana/
│       ├── analytics/    # Analytics database schema
│       ├── kafka-connect/ # Kafka Connect configuration
│       └── logging/
├── KEYCLOAK_SETUP.md      # Keycloak configuration guide
└── OBSERVABILITY_GUIDE.md # Monitoring and observability guide

Building Docker Images

Build images using JIB (recommended):

./deploy/docker/build-images-jib.sh

Or using Buildpacks:

./deploy/docker/build-images-buildpacks.sh

Running Tests

# Run all tests
mvn test

# Run tests for specific service
cd order && mvn test

🐳 Docker Deployment

Using Docker Compose

All services can be containerized and run via Docker Compose. See deploy/docker/docker-compose.yml for infrastructure services.

To build and run application services:

# Build images
./deploy/docker/build-images-jib.sh

# Run with docker-compose (add application services to compose file)
docker-compose up -d

🔒 Security

  • Authentication: OAuth2/OIDC via Keycloak
  • Authorization: Role-based access control (RBAC)
  • API Gateway: JWT token validation
  • Service-to-Service: Secure communication (configure TLS in production)

Roles

  • RESEARCHER: Can create reservations
  • PI: Can approve group member reservations
  • INSTRUMENT_OWNER: Can approve instrument reservations
  • ADMIN: Full system access
  • ANALYST: Can access analytics API and run queries

About

A production-ready microservices system built with Spring Boot and Spring Cloud for managing time-based reservations of shared scientific instruments (SEM/TEM, AFM, XRD, cryostats, lasers, cleanroom tools, GPU nodes, etc.) in research laboratories.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages