Skip to content

Latest commit

 

History

History
207 lines (153 loc) · 12.4 KB

deployment.md

File metadata and controls

207 lines (153 loc) · 12.4 KB

Deployment & Operations Guide

This document covers evaluation, API usage, runtime configuration, the SageMaker serving contract, AWS resources and IAM, deployment workflows, and operational constraints.

Retrieval Pipeline

The backend shallow-clones public repositories, excludes generated and dependency directories, and chunks supported files around functions, classes, methods, declarations, and containers. Large classes and source files also receive overview chunks, while plain-text and unsupported-language files use bounded fallback chunking. Clones are deleted after indexing.

Qdrant is both the vector store and persistent repository cache. A canonical repository-and-branch key lets a new session reuse an index. Re-indexing uploads a hidden generation, activates it only after every vector is stored, then removes the previous generation. A failed rebuild leaves the working cache intact.

Each question uses Qwen3 semantic retrieval, BM25, path and intent signals, and Reciprocal Rank Fusion. Candidates are deduplicated, reranked with Qwen3-Reranker-0.6B, and selected using relevance, source quality, and diversity signals.

Conversation Layer

Before retrieval, the conversation layer routes social messages, rewrites contextual follow-ups into standalone repository questions, and asks for clarification when a reference has no reliable antecedent. Debug responses can include a conversation_trace with the original query, standalone rewrite, retrieval query, and final Bedrock prompt. The answer prompt requires concrete files and symbols, evidence/inference separation, inline citations, and an explicit acknowledgement when evidence is insufficient. Citation numbers are validated before the response is returned.

Evaluation

The retrieval benchmark compares vector-only search with hybrid retrieval and reranking over 120 questions. See the evaluation guide for metric definitions, verified results, reproducibility details, and commands.

API

All repository operations are scoped by an X-Session-Id header.

Method Endpoint Purpose
POST /api/repos/index Reuse a cached index or asynchronously build one
GET /api/repos List repositories for the current session
GET /api/repos/{repo_id} Read indexing status and metadata
POST /api/query Ask a grounded question about an indexed repository
POST /api/session/end Clear session state; persistent vectors remain cached

Index a repository, reusing its cache by default:

curl -X POST http://localhost:8000/api/repos/index \
  -H 'Content-Type: application/json' \
  -H 'X-Session-Id: code-compass-demo-session' \
  -d '{
    "github_url": "https://github.com/fastapi/fastapi",
    "reindex": false
  }'

Set reindex to true to build and atomically activate a replacement generation.

Query the completed index:

curl -X POST http://localhost:8000/api/query \
  -H 'Content-Type: application/json' \
  -H 'X-Session-Id: code-compass-demo-session' \
  -d '{
    "repo_id": 1,
    "question": "Where is request validation handled?",
    "top_k": 8,
    "history": []
  }'

Runtime configuration

Variable Default Purpose
AWS_REGION us-east-1 fallback Amazon Bedrock region
APP_ENV local Runtime environment label
LOG_LEVEL INFO Application log level; dependency and Uvicorn access noise remains suppressed
ENABLE_PROFILING false Retain detailed timing data in process without emitting it to CloudWatch
PORT 8080 Direct container/listener port
BEDROCK_MODEL_ID qwen.qwen3-coder-next Bedrock model or inference profile
QDRANT_URL required Qdrant REST endpoint
QDRANT_API_KEY none Cloud key; omit only for an unsecured local instance
QDRANT_API_KEY_SECRET_ARN none Preferred AWS key source when direct key is absent
QDRANT_COLLECTION versioned default Application collection override
QDRANT_EVAL_COLLECTION versioned eval default Isolated evaluation collection
QDRANT_UPSERT_BATCH_SIZE 64 Indexing batch size
QDRANT_TIMEOUT_SECONDS 60 Client request timeout
CORS_ORIGINS http://localhost:3000 Comma-separated allowed origins
SESSION_TTL_MINUTES 120 Session lifetime
REPO_CACHE_DIR /opt/ml/codecompass/repos Writable clone directory
EMBEDDING_MODEL_ID Qwen/Qwen3-Embedding-0.6B Local embedding model
QWEN_EMBEDDING_BATCH_SIZE 8 Embedding batch size
RERANKER_MODEL_ID Qwen/Qwen3-Reranker-0.6B Local reranking model
RAG_RERANK_BATCH_SIZE 4 Reranker batch size
RAG_FINAL_SOURCE_LIMIT request top_k Maximum sources sent to generation
RAG_SEARCH_MULTIPLIER 4 Candidate depth for shallow questions
RAG_DEEP_SEARCH_MULTIPLIER 8 Candidate depth for implementation questions

SageMaker serving architecture

Browser
  │ HTTPS
  ▼
Vercel (React static site + same-origin serverless API adapter)
  │ Vercel OIDC → short-lived AWS credentials
  ▼
SageMaker Runtime InvokeEndpoint API
  │ POST /invocations on port 8080
  ▼
SageMaker real-time endpoint (one GPU instance, one FastAPI worker)
  ├── Qwen3 embedding + reranker models baked into the ECR image
  ├── Qdrant Cloud ── persistent vectors and repository cache
  ├── Amazon Bedrock ── grounded answer generation
  └── CloudWatch Logs ── container and endpoint logs

GitHub Actions (OIDC) ── build/test ──► ECR ──► SageMaker deployment

SageMaker endpoints are AWS APIs, not public general-purpose web servers. ui/api/proxy.js maps browser REST calls to /invocations, uses Vercel OIDC, and never sends AWS credentials to the browser. Direct /api/* routes remain available locally.

The container listens on 0.0.0.0:8080, responds to /ping, accepts /invocations, handles SageMaker's serve argument, and exits cleanly on SIGTERM. It reports healthy only after both local models and Qdrant are ready.

Required AWS resources

  • An ECR repository; scripts/push.sh creates it with scan-on-push if missing.
  • A SageMaker execution role trusted by sagemaker.amazonaws.com.
  • Real-time endpoint quota for the selected GPU instance. Scripts default to one ml.g5.xlarge; confirm regional memory and quota.
  • Bedrock access to BEDROCK_MODEL_ID in the same region.
  • A reachable Qdrant Cloud cluster and API key. The workflow reads QDRANT_API_KEY from a GitHub environment secret; QDRANT_API_KEY_SECRET_ARN offers stronger handling.
  • GitHub and Vercel OIDC identity providers with narrowly scoped roles.

SageMaker execution role

The runtime role needs:

  • ECR pull: ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage;
  • logging: logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents, logs:DescribeLogStreams, and cloudwatch:PutMetricData;
  • generation: bedrock:InvokeModel on the chosen model or inference-profile ARN;
  • optional secret loading: secretsmanager:GetSecretValue on the Qdrant secret ARN.

GitHub deployment role

The CI role needs the ECR push actions ecr:GetAuthorizationToken, ecr:CreateRepository, ecr:DescribeRepositories, ecr:BatchCheckLayerAvailability, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, and ecr:PutImage; SageMaker create, describe, update, and delete permissions for models, endpoint configurations, and the endpoint; and iam:PassRole restricted to the execution role with iam:PassedToService = sagemaker.amazonaws.com.

Restrict the GitHub OIDC sub claim to this repository, main, and the protected production environment.

Vercel runtime role

The role needs only sagemaker:InvokeEndpoint on this endpoint. Restrict its OIDC trust policy to the production Vercel project and environment.

Deploy from a workstation

Prerequisites

Install Docker, AWS CLI v2, jq, and openssl, and configure AWS credentials authorized for ECR and SageMaker.

Environment variables

Load AWS_REGION, ECR_REPOSITORY, SAGEMAKER_ENDPOINT_NAME, SAGEMAKER_EXECUTION_ROLE_ARN, QDRANT_URL, and CORS_ORIGINS into the current shell. Provide either QDRANT_API_KEY_SECRET_ARN or QDRANT_API_KEY. SAGEMAKER_INSTANCE_TYPE defaults to ml.g5.xlarge; the remaining runtime options are documented in Runtime configuration.

Deploy

./scripts/test-local-sagemaker.sh
./scripts/push.sh
./scripts/deploy.sh

test-local-sagemaker.sh builds the production image, runs it as the non-root production user on port 8080 with SageMaker-style writable mounts, polls the real GET /ping status until it receives 200, then checks /api/health and /invocations. It prints logs and fails on startup errors.

build.sh targets Linux/amd64 and preloads both Hugging Face snapshots. PRELOAD_MODELS=0 produces a faster development build that needs outbound Hugging Face access at startup and is not recommended for production. push.sh authenticates, tags, and pushes to ECR. deploy.sh hashes the image and environment into immutable model/config names, updates the stable endpoint only when necessary, waits for InService, and recreates a Failed endpoint.

The Dockerfile deliberately uses the official PyTorch CUDA runtime as a single stage. A Python builder stage would retain another copy of several gigabytes of CUDA/PyTorch wheels without shrinking the runtime layer. Baking both 0.6B models trades ECR storage and build time for predictable, network-independent startup. Runtime code is copied last for layer caching, caches are removed, and the service runs as a non-root user with only Git retained for cloning.

Old models and endpoint configurations are retained for rollback; delete unused versions periodically after validating a deployment.

GitHub Actions CI/CD

.github/workflows/deploy-sagemaker.yml runs unit and container-parity validation on every push and pull request. Main-branch pushes and manual runs also push the verified image and deploy it to SageMaker. Deployment jobs request id-token: write and use GitHub OIDC; no AWS access keys are stored.

Create a protected GitHub environment named production with these secrets:

Secret Purpose
AWS_GITHUB_ROLE_ARN OIDC deployment role
AWS_REGION ECR, SageMaker, Secrets Manager, and Bedrock region
ECR_REPOSITORY Backend repository name
SAGEMAKER_ENDPOINT_NAME Stable endpoint name
SAGEMAKER_EXECUTION_ROLE_ARN Runtime role passed to SageMaker
SAGEMAKER_INSTANCE_TYPE Normally ml.g5.xlarge
QDRANT_URL Qdrant Cloud HTTPS endpoint
QDRANT_API_KEY Qdrant key used by the workflow
CORS_ORIGINS Production Vercel origin
BEDROCK_MODEL_ID Bedrock model or inference profile

All values use GitHub's secrets context. Environment protection and required reviewers are recommended.

The direct Qdrant key is convenient for a personal deployment, but places it in the SageMaker model's container environment. AWS advises against sensitive values in CreateModel environment fields. For shared or long-lived deployments, set QDRANT_API_KEY_SECRET_ARN; the container retrieves it on startup and the execution role must allow secretsmanager:GetSecretValue on that ARN.

For Vercel, set AWS_ROLE_ARN, SAGEMAKER_AWS_REGION, and SAGEMAKER_ENDPOINT_NAME. Enable OIDC and configure the trust policy. Leave REACT_APP_API_URL unset in production so the UI uses the same-origin adapter; set it to http://localhost:8000 locally.

Known Operational Constraints

  • Only public GitHub repositories are supported.
  • Repository metadata, active sessions, and lexical indexes are process-local.
  • Indexing runs as an in-process background task and is interrupted by deployment or instance failure.
  • Real-time invocations have a 60-second response window.
  • Session IDs scope data but are not user authentication.
  • GitHub cloning, Qdrant, and Bedrock require network egress.
  • Large repositories take time to parse and embed.
  • Both local models benefit substantially from CUDA acceleration.
  • Model-based reranking is the largest retrieval-time cost.
  • The system does not build a call graph or dependency graph.
  • The Python dependency file is version-pinned but not hash-locked.
  • Create React App contributes known transitive npm audit findings.
  • A GPU endpoint accrues charges while it remains InService.