TESTING_GUIDE.md

Multi-Agent System Testing Guide

Complete guide for testing the multi-agent orchestration system.

Quick Start

# Install dependencies
pip install -r requirements.txt

# Run all tests
pytest

# Run with coverage
pytest --cov=multiagent --cov-report=html

# Run specific category
pytest -m integration
pytest -m load
pytest -m security

# Run load tests with Locust
locust -f tests/load/locustfile.py --host=http://localhost:8000

# Run security scan
./scripts/security_scan.sh

Test Categories

1. Integration Tests (27 tests)

Purpose: Validate complete system workflows and component interactions

Files:

• tests/integration/test_full_system.py - Complete lifecycle tests
• tests/integration/test_api_integration.py - API endpoint tests
• tests/integration/test_websocket_integration.py - WebSocket tests

Run:

pytest -m integration -v

Key Tests:

• Complete task lifecycle from creation to completion
• Concurrent task handling (10+ tasks simultaneously)
• Rate limiting enforcement
• Error recovery and graceful degradation
• Agent selection and coordination
• Progress tracking accuracy
• Cost tracking and optimization
• WebSocket real-time updates

Expected Results:

• All tasks complete successfully
• No resource leaks
• Correct state management
• Accurate metrics and tracking

2. Load Tests (10 tests)

Purpose: Validate performance, throughput, and resource usage

Files:

• tests/load/test_performance.py - Performance tests
• tests/load/locustfile.py - Locust load testing

Run pytest tests:

pytest -m load -v

Run Locust tests:

# Interactive mode
locust -f tests/load/locustfile.py --host=http://localhost:8000

# Headless mode (50 users, 5/sec spawn rate, 300s duration)
locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 50 -r 5 --headless -t 300s \
  --csv=reports/load_test

Performance Targets:

• Throughput: >10 tasks/second
• P50 Latency: <500ms
• P95 Latency: <2000ms
• P99 Latency: <5000ms
• Memory increase: <500MB under load
• CPU usage: Reasonable (system-dependent)

Locust User Types:

1. MultiAgentUser: Normal behavior (creates tasks, checks status)
2. HeavyUser: Rapid task creation (stress testing)
3. ReadOnlyUser: Monitoring/dashboard behavior
4. BurstUser: Burst traffic simulation

Key Metrics:

• Requests per second (RPS)
• Response time distribution
• Error rate
• Memory usage
• CPU utilization
• Database connection pool usage

3. Security Tests (15 tests)

Purpose: Validate security measures and prevent vulnerabilities

Files:

• tests/security/test_security.py - Security vulnerability tests
• scripts/security_scan.sh - Security scanning script

Run pytest tests:

pytest -m security -v

Run security scan:

./scripts/security_scan.sh

What's Tested:

• SQL Injection: Validates input sanitization
• XSS (Cross-Site Scripting): Tests HTML/JS escaping
• Rate Limiting: Ensures abuse prevention
• API Key Exposure: Checks no secrets in responses
• Input Validation: Tests type and format checking
• Information Disclosure: Ensures safe error messages
• Authentication Bypass: Validates auth enforcement
• Path Traversal: Prevents file system access
• Command Injection: Blocks shell command execution
• DDoS Protection: Tests request size and rate limits
• CORS Policy: Validates cross-origin settings
• Security Headers: Checks HTTP security headers
• Sensitive Data: Ensures no PII/credentials exposed

Security Tools:

• Bandit: Python security linter
• Safety: Dependency vulnerability checker
• pip-audit: Package vulnerability scanner
• Semgrep: Static analysis (optional)
• Trivy: Container security scanner (optional)

View Reports:

cat reports/security_summary.txt
cat reports/bandit_report.txt
cat reports/safety_report.txt

Test Execution

Running Tests

All tests:

pytest

Specific file:

pytest tests/integration/test_full_system.py

Specific test:

pytest tests/integration/test_full_system.py::test_complete_task_lifecycle

With markers:

pytest -m integration
pytest -m "integration and not slow"
pytest -m "load or security"

Parallel execution:

pytest -n auto  # Auto-detect CPUs
pytest -n 4     # Use 4 workers

With coverage:

pytest --cov=multiagent --cov-report=html --cov-report=term-missing

Verbose output:

pytest -v      # Verbose
pytest -vv     # Very verbose
pytest -s      # Show print statements

Test Configuration

pytest.ini contains:

• Test discovery patterns
• Default options
• Test markers
• Coverage configuration
• Logging settings
• Timeout configuration

conftest.py provides:

• Shared fixtures
• Test configuration
• Setup/teardown hooks
• Custom markers

Test Markers

Available markers:

• @pytest.mark.integration - Integration tests
• @pytest.mark.load - Load/performance tests
• @pytest.mark.security - Security tests
• @pytest.mark.slow - Slow tests (>5 seconds)
• @pytest.mark.api - API endpoint tests
• @pytest.mark.websocket - WebSocket tests
• @pytest.mark.database - Database tests
• @pytest.mark.redis - Redis tests

Coverage

Running with Coverage

# Run tests with coverage
pytest --cov=multiagent --cov-report=html

# View HTML report
open reports/coverage/index.html

# Terminal report with missing lines
pytest --cov=multiagent --cov-report=term-missing

# XML report (for CI/CD)
pytest --cov=multiagent --cov-report=xml

Coverage Goals

• Overall: >80% coverage
• Critical paths: >90% coverage
• New code: 100% coverage
• Excluding: Tests, migrations, generated code

Coverage Reports

Reports are generated in reports/:

• reports/coverage/index.html - HTML report
• reports/coverage.xml - XML report (for CI/CD)
• Terminal output shows missing lines

Load Testing Details

Using Locust

1. Start Locust Web UI:

locust -f tests/load/locustfile.py --host=http://localhost:8000

Open http://localhost:8089

2. Configure Test:

• Number of users (total)
• Spawn rate (users per second)
• Host URL

3. Start Test: Click "Start swarming"

4. Monitor:

• Charts show RPS, response times
• Statistics table shows request metrics
• Failures tab shows errors

5. Stop Test: Click "Stop"

6. Download Report: Click "Download Data" for CSV

Headless Mode (CI/CD)

# Basic test
locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 10 -r 2 \
  --headless -t 60s

# Heavy load
locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 100 -r 10 \
  --headless -t 300s

# Export results
locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 50 -r 5 \
  --headless -t 300s \
  --csv=reports/load_test \
  --html=reports/load_test.html

Load Test Scenarios

1. Smoke Test (Quick validation):

locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 5 -r 1 --headless -t 30s

2. Load Test (Normal load):

locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 50 -r 5 --headless -t 300s

3. Stress Test (High load):

locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 200 -r 20 --headless -t 600s

4. Spike Test (Sudden burst):

locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 1000 -r 100 --headless -t 60s

5. Endurance Test (Long duration):

locust -f tests/load/locustfile.py \
  --host=http://localhost:8000 \
  -u 50 -r 5 --headless -t 3600s

Security Testing Details

Running Security Scan

# Full scan
./scripts/security_scan.sh

# View summary
cat reports/security_summary.txt

# View individual reports
cat reports/bandit_report.txt
cat reports/safety_report.txt
cat reports/pip_audit_report.txt

Individual Tools

Bandit (Python security):

bandit -r multiagent/ -f json -o reports/bandit_report.json
bandit -r multiagent/  # Terminal output

Safety (Dependencies):

safety check
safety check --json > reports/safety_report.json

pip-audit (Packages):

pip-audit
pip-audit --format=json > reports/pip_audit_report.json

Semgrep (Static analysis):

semgrep --config=auto multiagent/

Security Test Categories

1. Injection Attacks:

   • SQL injection
   • Command injection
   • Path traversal

2. Cross-Site Attacks:

   • XSS (Cross-Site Scripting)
   • CSRF (Cross-Site Request Forgery)

3. Authentication & Authorization:

   • Authentication bypass
   • Token validation
   • Permission checks

4. Data Protection:

   • API key exposure
   • Sensitive data in responses
   • Information disclosure

5. Resource Protection:

   • Rate limiting
   • DDoS protection
   • Request size limits

6. Network Security:

   • CORS policy
   • Security headers
   • TLS/SSL configuration

CI/CD Integration

GitHub Actions

Example .github/workflows/tests.yml:

name: Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: |
          pytest --cov=multiagent --cov-report=xml

      - name: Run security scan
        run: ./scripts/security_scan.sh

      - name: Upload coverage
        uses: codecov/codecov-action@v2
        with:
          file: ./reports/coverage.xml

Pre-commit Hooks

.pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: pytest
        name: pytest
        entry: pytest tests/
        language: system
        pass_filenames: false

Troubleshooting

Common Issues

1. Import Errors:

# Add project to PYTHONPATH
export PYTHONPATH=$PYTHONPATH:$(pwd)

# Or install in editable mode
pip install -e .

2. Async Test Failures:

• Ensure @pytest.mark.asyncio decorator
• Check asyncio_mode = auto in pytest.ini

3. Timeout Errors:

# Increase timeout
pytest --timeout=600

# Or per-test
@pytest.mark.timeout(600)

4. Database Connection Errors:

• Ensure database is running
• Check connection string
• Verify credentials

5. WebSocket Connection Errors:

• Ensure API server is running
• Check WebSocket URL
• Verify WebSocket support in test client

Debugging

Run single test:

pytest tests/integration/test_full_system.py::test_complete_task_lifecycle -v

Show print statements:

pytest -s

Drop into debugger:

pytest --pdb  # On failure
pytest --pdb --maxfail=1  # Stop on first failure

Show locals:

pytest -l

Detailed traceback:

pytest --tb=long

Best Practices

Writing Tests

1. Independent: Each test should be independent
2. Fast: Keep tests fast (<1s for unit tests)
3. Clear: Use descriptive names and docstrings
4. Isolated: Mock external dependencies
5. Comprehensive: Test happy path and edge cases
6. Cleanup: Always clean up resources

Test Structure

@pytest.mark.integration
@pytest.mark.asyncio
async def test_feature_name():
    """
    Brief description

    Validates:
    1. First thing
    2. Second thing
    3. Third thing
    """
    # Arrange
    setup_data = create_test_data()

    # Act
    result = await perform_action(setup_data)

    # Assert
    assert result == expected, "Failure message"

Fixtures

Use fixtures for common setup:

@pytest.fixture
def sample_data():
    """Provide sample test data"""
    return {"key": "value"}

def test_with_fixture(sample_data):
    """Test using fixture"""
    assert sample_data["key"] == "value"

Continuous Improvement

Regular Testing

• Run tests before committing
• Run full suite before merging
• Monitor coverage trends
• Review failed tests promptly

Test Maintenance

• Keep tests up to date with code changes
• Remove obsolete tests
• Refactor duplicated test code
• Update test documentation

Quality Metrics

Track:

• Test count
• Code coverage percentage
• Test execution time
• Failure rate
• Flaky test count

Resources

• pytest documentation
• Locust documentation
• Coverage.py documentation
• OWASP Testing Guide

Support

For questions:

• Check test examples in tests/
• Review pytest documentation
• Check existing test output
• Contact development team

---

Last Updated: October 2025 Bot 5: Testing & Quality Assurance