A FastAPI-based queue server that provides REST API endpoints for message queuing operations, compatible with the PyQueue client library. Features configurable storage backends for flexibility and performance.
When to use it: you want SQS-style semantics (visibility timeouts, receipt handles, at-least-once delivery) on your own server, without running Redis, RabbitMQ or an AWS account. It fits small services, background jobs and homelab automation; for very high throughput or complex routing, a dedicated broker is a better choice.
- RESTful API for queue operations with comprehensive security
- Multiple storage backends: JSON files (lightweight) or SQLite (high performance)
- Multiple queue support with isolated message storage
- SQS-like message handling with visibility timeouts and receipt handles
- API Key authentication with queue-specific permissions
- Role-based access control (READ, WRITE, DELETE, MANAGE permissions)
- Health check endpoints for monitoring
- CORS support for web applications
- Docker support for easy deployment
PyQueue Server supports two storage backends that can be configured without code changes:
- Use case: Development, small deployments, simple setups
- Features: Human-readable files, easy debugging, minimal dependencies
- Performance: Good for < 10,000 messages per queue
- File location:
./data/*.json(one file per queue)
- Use case: Production, high-performance, concurrent access
- Features: ACID transactions, better concurrency, optimized queries
- Performance: Excellent for > 10,000 messages per queue
- Database location:
./data/pyqueue.db(single database file)
Simply change the STORAGE_BACKEND setting in your .env file:
# Use JSON storage (default)
STORAGE_BACKEND=json
# Use SQLite storage
STORAGE_BACKEND=sqliteNo code changes required - the server automatically uses the configured backend!
POST /api/v1/queues/{queue_name}/messages- Add message to queueGET /api/v1/queues/{queue_name}/messages- Get messages from queue (non-destructive)POST /api/v1/queues/{queue_name}/messages/receive- Receive messages (SQS-style with visibility timeout)DELETE /api/v1/queues/{queue_name}/messages/{receipt_handle}- Delete message by receipt handleDELETE /api/v1/queues/{queue_name}/messages/by-id/{message_id}- Delete message by IDPUT /api/v1/queues/{queue_name}/messages/by-id/{message_id}- Update message contentDELETE /api/v1/queues/{queue_name}/messages- Clear all messages from queue
GET /api/v1/queues/{queue_name}/info- Get queue statistics and informationGET /api/v1/queues/{queue_name}/health- Health check for specific queueGET /api/v1/queues- List all accessible queuesGET /api/v1/health- Global health check
GET /api/v1/auth/me- Get current API key informationGET /api/v1/auth/permissions/{queue_name}- Check permissions for specific queue
🔐 Note: All queue operations require API key authentication via X-API-Key header.
-
Clone or create the project directory
-
Install dependencies:
pip install -r requirements.txt
-
Set up API keys (Required for v1.0+):
Copy the example and replace its keys with your own (
config/api_keys.jsonis not tracked by git, so your keys stay out of the repository):cp config/api_keys.example.json config/api_keys.json
Use long random strings as keys, e.g.
python -c "import secrets; print('pk_' + secrets.token_urlsafe(24))". A minimal file looks like this:{ "your-api-key-here": { "description": "Development API Key", "queues": { "*": ["READ", "WRITE", "DELETE", "MANAGE"] } } } -
Configure storage backend (Optional):
Create
.envfile to customize settings:# Choose storage backend STORAGE_BACKEND=json # or "sqlite" # Storage paths JSON_STORAGE_DIR=./data SQLITE_DB_PATH=./data/pyqueue.db # Server settings HOST=localhost PORT=8000 DEBUG=true
python main.py# Development mode (with auto-reload)
python start_server.py
# Production mode
python start_server.py --production --host 0.0.0.0
# Custom configuration
python start_server.py --host 0.0.0.0 --port 9000 --log-level DEBUG# Development
uvicorn main:app --reload --host localhost --port 8000
# Production
uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4# Build and run with Docker Compose
docker-compose up --build
# Or build and run manually
docker build -t pyqueue-server .
docker run -p 8002:8002 -v $(pwd)/config:/app/config pyqueue-serverThe server will start on http://localhost:8000 (Docker image and Docker Compose: port 8002)
API Documentation: Visit http://localhost:8000/docs for interactive API documentation
First, install the PyQueue client library:
pip install pyqueue_client==0.3.0Then use it in your applications:
from pyqueue_client import PyQueue
# Create remote queue client
queue = PyQueue(
queue_type="remote",
server_url="http://localhost:8000",
queue_name="my_queue",
api_key="your-api-key-here" # Required for authentication
)
# Add a message
queue.add_message({
"type": "user_signup",
"user_id": "123",
"email": "user@example.com"
})
# Get queue information
info = queue.get_queue_info()
print(f"Messages in queue: {info['message_count']}")from pyqueue_client import PyQueue
import time
# Create consumer client
consumer = PyQueue(
queue_type="remote",
server_url="http://localhost:8000",
queue_name="my_queue",
api_key="your-api-key-here" # Required for authentication
) queue_type="remote",
server_url="http://localhost:8000",
queue_name="my_queue",
api_key="your-api-key-here" # Required for authentication
)
while True:
# Receive messages (SQS-style)
messages = consumer.receive_messages(max_messages=5, visibility_timeout=30)
for message in messages:
try:
# Process your message
print(f"Processing: {message['message_body']}")
# Delete message after successful processing
consumer.delete_message(message['receipt_handle'])
except Exception as e:
print(f"Error processing message: {e}")
if not messages:
time.sleep(5) # Wait before checking againCreate a .env file to configure PyQueue Server:
# ===== STORAGE CONFIGURATION =====
# Choose your storage backend
STORAGE_BACKEND=json # "json" or "sqlite"
# JSON storage settings (when STORAGE_BACKEND=json)
JSON_STORAGE_DIR=./data
# SQLite storage settings (when STORAGE_BACKEND=sqlite)
SQLITE_DB_PATH=./data/pyqueue.db
# ===== SERVER CONFIGURATION =====
HOST=localhost # Server bind address
PORT=8000 # Server port
DEBUG=true # Enable debug mode
LOG_LEVEL=INFO # Logging level (DEBUG, INFO, WARNING, ERROR)
# ===== QUEUE CONFIGURATION =====
QUEUE_DATA_DIR=./data # Legacy compatibility
MAX_MESSAGE_SIZE=262144 # Maximum message size in bytes (256KB)
DEFAULT_VISIBILITY_TIMEOUT=30 # Default visibility timeout in seconds
MAX_RECEIVE_COUNT=10 # Maximum receive count per message
# ===== API CONFIGURATION =====
API_V1_PREFIX=/api/v1 # API endpoint prefix| Feature | JSON Storage | SQLite Storage |
|---|---|---|
| Setup | Zero configuration | Zero configuration |
| Performance | Good (< 10K msgs) | Excellent (> 10K msgs) |
| Concurrency | File locking | Database transactions |
| Debugging | Human readable files | SQL queries |
| Backup | Copy JSON files | Single database file |
| Dependencies | None | aiosqlite |
| Use Case | Development, small scale | Production, high volume |
Create config/api_keys.json (start from config/api_keys.example.json):
{
"dev-key-123": {
"description": "Development API Key",
"queues": {
"*": ["READ", "WRITE", "DELETE", "MANAGE"]
}
},
"prod-consumer-456": {
"description": "Production Consumer",
"queues": {
"orders": ["READ", "DELETE"],
"notifications": ["READ", "DELETE"]
}
},
"prod-publisher-789": {
"description": "Production Publisher",
"queues": {
"orders": ["WRITE"],
"notifications": ["WRITE"]
}
}
}- READ: Get messages, receive messages, view queue info
- WRITE: Add messages, update messages
- DELETE: Delete individual messages
- MANAGE: Clear entire queues, full queue management
pyqueue-server/
├── main.py # FastAPI application entry point
├── requirements.txt # Python dependencies
├── Dockerfile # Docker configuration
├── docker-compose.yml # Docker Compose configuration
├── .env # Environment configuration
├── SECURITY.md # Security documentation
├── test_server.py # Server functionality tests
├── test_security.py # Security system tests
├── test_storage_backends.py # Storage backend tests
├── config/
│ ├── api_keys.example.json # Example API key configuration
│ └── api_keys.json # Your API keys (not tracked by git)
├── data/ # Queue data storage
│ ├── *.json # JSON queue files (if using JSON backend)
│ └── pyqueue.db # SQLite database (if using SQLite backend)
└── app/
├── core/
│ ├── config.py # Application configuration
│ ├── security.py # Authentication & authorization
│ └── storage/ # Storage abstraction layer
│ ├── __init__.py # Storage factory
│ ├── base.py # Storage interface
│ ├── json_storage.py # JSON file storage
│ └── sqlite_storage.py # SQLite storage
├── models/
│ └── queue.py # Pydantic data models
├── services/
│ └── queue_service.py # Queue management logic
└── api/
└── routes.py # REST API endpoints
uvicorn main:app --reload --host 0.0.0.0 --port 8000Run the comprehensive test suite:
# Test server functionality
python test_server.py
# Test security system
python test_security.py
# Test both storage backends
python test_storage_backends.pyNote: All API calls require the X-API-Key header for authentication.
# Health check (no authentication required)
curl http://localhost:8000/health
# Add a message
curl -X POST http://localhost:8000/api/v1/queues/test/messages \
-H "Content-Type: application/json" \
-H "X-API-Key: your-api-key-here" \
-d '{"message": {"content": "Hello World"}}'
# Get messages
curl -H "X-API-Key: your-api-key-here" \
http://localhost:8000/api/v1/queues/test/messages
# Receive messages (SQS-style)
curl -X POST -H "X-API-Key: your-api-key-here" \
"http://localhost:8000/api/v1/queues/test/messages/receive?max_messages=5&visibility_timeout=30"
# Queue info
curl -H "X-API-Key: your-api-key-here" \
http://localhost:8000/api/v1/queues/test/info
# List accessible queues
curl -H "X-API-Key: your-api-key-here" \
http://localhost:8000/api/v1/queues
# Check API key permissions
curl -H "X-API-Key: your-api-key-here" \
http://localhost:8000/api/v1/auth/meTest SQLite vs JSON storage performance:
# Test with JSON storage
STORAGE_BACKEND=json python test_storage_backends.py
# Test with SQLite storage
STORAGE_BACKEND=sqlite python test_storage_backends.pycurl -H "X-API-Key: your-api-key-here"
http://localhost:8000/api/v1/queues/test/messages
curl -H "X-API-Key: your-api-key-here"
http://localhost:8000/api/v1/queues/test/info
### Running Tests
```bash
# Run all tests
python -m pytest tests/
# Run specific test suites
python test_server.py # Core functionality
python test_security.py # Authentication & authorization
python test_storage_backends.py # Storage backend comparison
-
Backup your data:
cp -r ./data ./data_backup
-
Update configuration:
# In .env file STORAGE_BACKEND=sqlite -
Restart the server:
python main.py
-
The SQLite database will be created automatically
-
Update configuration:
# In .env file STORAGE_BACKEND=json -
Restart the server:
python main.py
-
Existing JSON files will be used automatically
Note: Data migration between backends is not automatic. Each backend maintains its own data store.
# Build and run with Docker Compose
docker-compose up --build
# Run in background
docker-compose up -d --build
# View logs
docker-compose logs -f
# Stop services
docker-compose down# Build image
docker build -t pyqueue-server .
# Run with JSON storage
docker run -p 8002:8002 \
-v $(pwd)/data:/app/data \
-v $(pwd)/config:/app/config \
pyqueue-server
# Run with SQLite storage
docker run -p 8002:8002 \
-e STORAGE_BACKEND=sqlite \
-v $(pwd)/data:/app/data \
-v $(pwd)/config:/app/config \
pyqueue-server# docker-compose.yml
version: '3.8'
services:
pyqueue-server:
build: .
ports:
- "8002:8002"
environment:
- STORAGE_BACKEND=sqlite
- HOST=0.0.0.0
- PORT=8002
- DEBUG=false
- LOG_LEVEL=INFO
volumes:
- ./data:/app/data
- ./config:/app/config# Global health check
curl http://localhost:8000/health
# Queue-specific health check
curl -H "X-API-Key: your-api-key" \
http://localhost:8000/api/v1/queues/your-queue/healthMonitor these endpoints for production deployment:
GET /health- Overall service healthGET /api/v1/queues- Queue statisticsGET /api/v1/queues/{name}/info- Individual queue metrics
- Use SQLite storage for better performance
- Set up proper API keys with specific permissions
- Configure appropriate visibility timeouts
- Monitor queue depths and processing rates
- Set up log aggregation for debugging
- Use Docker for consistent deployments
- Configurable Storage Backends: Choose between JSON files or SQLite database
- API Key Authentication: Secure access with configurable API keys
- Role-Based Permissions: Fine-grained access control per queue
- Enhanced Performance: SQLite backend for high-throughput scenarios
- Storage Abstraction: Switch backends without code changes
- Async Storage Operations: All storage operations are fully asynchronous
- Better Error Handling: Comprehensive error responses and logging
- Health Check System: Monitor both global and queue-specific health
- Type Safety: Full type hints throughout the codebase
- Comprehensive Testing: Test suites for functionality, security, and storage
- SQLite Integration: Database transactions for better concurrency
- Optimized Queries: Indexed database operations for fast lookups
- Connection Pooling: Efficient database connection management
- Background Processing: Non-blocking I/O operations
| Project | What it is |
|---|---|
| pyqueue-server | Self-hosted queue server with an AWS SQS-style REST API (FastAPI, SQLite or JSON) |
| pyqueue_client | Python client library (pip install pyqueue-client) for a local JSON queue or a remote server |
| pyqueue-ui | Web dashboard to browse queues and messages |
| pynotiq | Worker that delivers queue messages as Telegram notifications |
| keystorebot | Telegram bot that stores key-value pairs in a PyQueue queue |
MIT License
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests for new functionality
- Ensure all tests pass
- Submit a pull request
- Documentation: Check the
/docsendpoint when the server is running - Issues: Report bugs and feature requests on GitHub
- Security: See
SECURITY.mdfor security-related information