This document covers evaluation, API usage, runtime configuration, the SageMaker serving contract, AWS resources and IAM, deployment workflows, and operational constraints.
The backend shallow-clones public repositories, excludes generated and dependency directories, and chunks supported files around functions, classes, methods, declarations, and containers. Large classes and source files also receive overview chunks, while plain-text and unsupported-language files use bounded fallback chunking. Clones are deleted after indexing.
Qdrant is both the vector store and persistent repository cache. A canonical repository-and-branch key lets a new session reuse an index. Re-indexing uploads a hidden generation, activates it only after every vector is stored, then removes the previous generation. A failed rebuild leaves the working cache intact.
Each question uses Qwen3 semantic retrieval, BM25, path and intent signals, and Reciprocal Rank Fusion. Candidates are deduplicated, reranked with Qwen3-Reranker-0.6B, and selected using relevance, source quality, and diversity signals.
Before retrieval, the conversation layer routes social messages, rewrites contextual follow-ups into standalone repository questions, and asks for clarification when a reference has no reliable antecedent. Debug responses can include a conversation_trace with the original query, standalone rewrite, retrieval query, and final Bedrock prompt. The answer prompt requires concrete files and symbols, evidence/inference separation, inline citations, and an explicit acknowledgement when evidence is insufficient. Citation numbers are validated before the response is returned.
The retrieval benchmark compares vector-only search with hybrid retrieval and reranking over 120 questions. See the evaluation guide for metric definitions, verified results, reproducibility details, and commands.
All repository operations are scoped by an X-Session-Id header.
| Method | Endpoint | Purpose |
|---|---|---|
POST |
/api/repos/index |
Reuse a cached index or asynchronously build one |
GET |
/api/repos |
List repositories for the current session |
GET |
/api/repos/{repo_id} |
Read indexing status and metadata |
POST |
/api/query |
Ask a grounded question about an indexed repository |
POST |
/api/session/end |
Clear session state; persistent vectors remain cached |
Index a repository, reusing its cache by default:
curl -X POST http://localhost:8000/api/repos/index \
-H 'Content-Type: application/json' \
-H 'X-Session-Id: code-compass-demo-session' \
-d '{
"github_url": "https://github.com/fastapi/fastapi",
"reindex": false
}'Set reindex to true to build and atomically activate a replacement generation.
Query the completed index:
curl -X POST http://localhost:8000/api/query \
-H 'Content-Type: application/json' \
-H 'X-Session-Id: code-compass-demo-session' \
-d '{
"repo_id": 1,
"question": "Where is request validation handled?",
"top_k": 8,
"history": []
}'| Variable | Default | Purpose |
|---|---|---|
AWS_REGION |
us-east-1 fallback |
Amazon Bedrock region |
APP_ENV |
local |
Runtime environment label |
LOG_LEVEL |
INFO |
Application log level; dependency and Uvicorn access noise remains suppressed |
ENABLE_PROFILING |
false |
Retain detailed timing data in process without emitting it to CloudWatch |
PORT |
8080 |
Direct container/listener port |
BEDROCK_MODEL_ID |
qwen.qwen3-coder-next |
Bedrock model or inference profile |
QDRANT_URL |
required | Qdrant REST endpoint |
QDRANT_API_KEY |
none | Cloud key; omit only for an unsecured local instance |
QDRANT_API_KEY_SECRET_ARN |
none | Preferred AWS key source when direct key is absent |
QDRANT_COLLECTION |
versioned default | Application collection override |
QDRANT_EVAL_COLLECTION |
versioned eval default | Isolated evaluation collection |
QDRANT_UPSERT_BATCH_SIZE |
64 |
Indexing batch size |
QDRANT_TIMEOUT_SECONDS |
60 |
Client request timeout |
CORS_ORIGINS |
http://localhost:3000 |
Comma-separated allowed origins |
SESSION_TTL_MINUTES |
120 |
Session lifetime |
REPO_CACHE_DIR |
/opt/ml/codecompass/repos |
Writable clone directory |
EMBEDDING_MODEL_ID |
Qwen/Qwen3-Embedding-0.6B |
Local embedding model |
QWEN_EMBEDDING_BATCH_SIZE |
8 |
Embedding batch size |
RERANKER_MODEL_ID |
Qwen/Qwen3-Reranker-0.6B |
Local reranking model |
RAG_RERANK_BATCH_SIZE |
4 |
Reranker batch size |
RAG_FINAL_SOURCE_LIMIT |
request top_k |
Maximum sources sent to generation |
RAG_SEARCH_MULTIPLIER |
4 |
Candidate depth for shallow questions |
RAG_DEEP_SEARCH_MULTIPLIER |
8 |
Candidate depth for implementation questions |
Browser
│ HTTPS
▼
Vercel (React static site + same-origin serverless API adapter)
│ Vercel OIDC → short-lived AWS credentials
▼
SageMaker Runtime InvokeEndpoint API
│ POST /invocations on port 8080
▼
SageMaker real-time endpoint (one GPU instance, one FastAPI worker)
├── Qwen3 embedding + reranker models baked into the ECR image
├── Qdrant Cloud ── persistent vectors and repository cache
├── Amazon Bedrock ── grounded answer generation
└── CloudWatch Logs ── container and endpoint logs
GitHub Actions (OIDC) ── build/test ──► ECR ──► SageMaker deployment
SageMaker endpoints are AWS APIs, not public general-purpose web servers. ui/api/proxy.js maps browser REST calls to /invocations, uses Vercel OIDC, and never sends AWS credentials to the browser. Direct /api/* routes remain available locally.
The container listens on 0.0.0.0:8080, responds to /ping, accepts /invocations, handles SageMaker's serve argument, and exits cleanly on SIGTERM. It reports healthy only after both local models and Qdrant are ready.
- An ECR repository;
scripts/push.shcreates it with scan-on-push if missing. - A SageMaker execution role trusted by
sagemaker.amazonaws.com. - Real-time endpoint quota for the selected GPU instance. Scripts default to one
ml.g5.xlarge; confirm regional memory and quota. - Bedrock access to
BEDROCK_MODEL_IDin the same region. - A reachable Qdrant Cloud cluster and API key. The workflow reads
QDRANT_API_KEYfrom a GitHub environment secret;QDRANT_API_KEY_SECRET_ARNoffers stronger handling. - GitHub and Vercel OIDC identity providers with narrowly scoped roles.
The runtime role needs:
- ECR pull:
ecr:GetAuthorizationToken,ecr:BatchCheckLayerAvailability,ecr:GetDownloadUrlForLayer, andecr:BatchGetImage; - logging:
logs:CreateLogGroup,logs:CreateLogStream,logs:PutLogEvents,logs:DescribeLogStreams, andcloudwatch:PutMetricData; - generation:
bedrock:InvokeModelon the chosen model or inference-profile ARN; - optional secret loading:
secretsmanager:GetSecretValueon the Qdrant secret ARN.
The CI role needs the ECR push actions ecr:GetAuthorizationToken, ecr:CreateRepository, ecr:DescribeRepositories, ecr:BatchCheckLayerAvailability, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, and ecr:PutImage; SageMaker create, describe, update, and delete permissions for models, endpoint configurations, and the endpoint; and iam:PassRole restricted to the execution role with iam:PassedToService = sagemaker.amazonaws.com.
Restrict the GitHub OIDC sub claim to this repository, main, and the protected production environment.
The role needs only sagemaker:InvokeEndpoint on this endpoint. Restrict its OIDC trust policy to the production Vercel project and environment.
Install Docker, AWS CLI v2, jq, and openssl, and configure AWS credentials authorized for ECR and SageMaker.
Load AWS_REGION, ECR_REPOSITORY, SAGEMAKER_ENDPOINT_NAME, SAGEMAKER_EXECUTION_ROLE_ARN, QDRANT_URL, and CORS_ORIGINS into the current shell. Provide either QDRANT_API_KEY_SECRET_ARN or QDRANT_API_KEY. SAGEMAKER_INSTANCE_TYPE defaults to ml.g5.xlarge; the remaining runtime options are documented in Runtime configuration.
./scripts/test-local-sagemaker.sh
./scripts/push.sh
./scripts/deploy.shtest-local-sagemaker.sh builds the production image, runs it as the non-root production user on port 8080 with SageMaker-style writable mounts, polls the real GET /ping status until it receives 200, then checks /api/health and /invocations. It prints logs and fails on startup errors.
build.sh targets Linux/amd64 and preloads both Hugging Face snapshots. PRELOAD_MODELS=0 produces a faster development build that needs outbound Hugging Face access at startup and is not recommended for production. push.sh authenticates, tags, and pushes to ECR. deploy.sh hashes the image and environment into immutable model/config names, updates the stable endpoint only when necessary, waits for InService, and recreates a Failed endpoint.
The Dockerfile deliberately uses the official PyTorch CUDA runtime as a single stage. A Python builder stage would retain another copy of several gigabytes of CUDA/PyTorch wheels without shrinking the runtime layer. Baking both 0.6B models trades ECR storage and build time for predictable, network-independent startup. Runtime code is copied last for layer caching, caches are removed, and the service runs as a non-root user with only Git retained for cloning.
Old models and endpoint configurations are retained for rollback; delete unused versions periodically after validating a deployment.
.github/workflows/deploy-sagemaker.yml runs unit and container-parity validation on every push and pull request. Main-branch pushes and manual runs also push the verified image and deploy it to SageMaker. Deployment jobs request id-token: write and use GitHub OIDC; no AWS access keys are stored.
Create a protected GitHub environment named production with these secrets:
| Secret | Purpose |
|---|---|
AWS_GITHUB_ROLE_ARN |
OIDC deployment role |
AWS_REGION |
ECR, SageMaker, Secrets Manager, and Bedrock region |
ECR_REPOSITORY |
Backend repository name |
SAGEMAKER_ENDPOINT_NAME |
Stable endpoint name |
SAGEMAKER_EXECUTION_ROLE_ARN |
Runtime role passed to SageMaker |
SAGEMAKER_INSTANCE_TYPE |
Normally ml.g5.xlarge |
QDRANT_URL |
Qdrant Cloud HTTPS endpoint |
QDRANT_API_KEY |
Qdrant key used by the workflow |
CORS_ORIGINS |
Production Vercel origin |
BEDROCK_MODEL_ID |
Bedrock model or inference profile |
All values use GitHub's secrets context. Environment protection and required reviewers are recommended.
The direct Qdrant key is convenient for a personal deployment, but places it in the SageMaker model's container environment. AWS advises against sensitive values in CreateModel environment fields. For shared or long-lived deployments, set QDRANT_API_KEY_SECRET_ARN; the container retrieves it on startup and the execution role must allow secretsmanager:GetSecretValue on that ARN.
For Vercel, set AWS_ROLE_ARN, SAGEMAKER_AWS_REGION, and SAGEMAKER_ENDPOINT_NAME. Enable OIDC and configure the trust policy. Leave REACT_APP_API_URL unset in production so the UI uses the same-origin adapter; set it to http://localhost:8000 locally.
- Only public GitHub repositories are supported.
- Repository metadata, active sessions, and lexical indexes are process-local.
- Indexing runs as an in-process background task and is interrupted by deployment or instance failure.
- Real-time invocations have a 60-second response window.
- Session IDs scope data but are not user authentication.
- GitHub cloning, Qdrant, and Bedrock require network egress.
- Large repositories take time to parse and embed.
- Both local models benefit substantially from CUDA acceleration.
- Model-based reranking is the largest retrieval-time cost.
- The system does not build a call graph or dependency graph.
- The Python dependency file is version-pinned but not hash-locked.
- Create React App contributes known transitive
npm auditfindings. - A GPU endpoint accrues charges while it remains
InService.