ValetUpload: Efficient Multipart File Upload Service

This project provides a straightforward API service that simplifies large file uploads by handling the complexities of multipart uploads to AWS S3. It helps you manage file uploads from clients without them needing direct access to your S3 bucket or deep knowledge of S3's multipart upload process. You get a controlled, secure, and resilient way for users to upload files in chunks, especially useful for unreliable networks or very large files.

Installation

Getting ValetUpload up and running locally is pretty simple.

1. Clone the Repository

   Start by cloning the project to your local machine:

   git clone https://github.com/DanielPopoola/valet-upload.git
   cd valet-upload

2. Build the Project

   This project uses Maven. You'll need Java 21 or newer installed. Build the "fat jar" which includes all dependencies:

   mvn clean install

3. Environment Variables

   The service relies on a few environment variables for configuration. Create a .env file or set these in your shell:

   # Redis URL for tracking upload parts (e.g., redis://localhost:6379)
   REDIS_URL=redis://localhost:6379
   
   # AWS region for S3 operations (e.g., us-east-1)
   AWS_REGION=us-east-1
   
   # Name of the S3 bucket where files will be stored
   S3_BUCKET_NAME=your-s3-bucket-name
   
   # Port for the Javalin API server to listen on
   PORT=7000
   
   # Duration in seconds for which an incomplete upload's state is tracked in Redis
   UPLOAD_EXPIRATION_SECONDS=7200 # e.g., 2 hours
   
   # Duration in seconds for which S3 presigned URLs are valid
   PRESIGNED_URL_EXPIRATION_SECONDS=900 # e.g., 15 minutes

   You'll also need AWS credentials configured for the service to interact with S3. This typically involves setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or using IAM roles if deploying to an EC2 instance or ECS/EKS.

4. Run the Application

   Once built and configured, you can run the application:

   java -jar target/valet-upload-1.0-SNAPSHOT.jar

   The API will then be accessible on the configured PORT (default: 7000).

Usage

ValetUpload orchestrates the entire multipart upload process. Here's how a typical client-side upload flow would work:

1. Initiate an Upload: The client requests a new multipart upload by providing the desired filename and the total number of parts the file will be split into.
2. Get Presigned URLs: For each part, the client requests a presigned URL from ValetUpload. These URLs allow the client to upload data directly to S3 without needing AWS credentials.
3. Upload Parts: The client uploads each file part to its respective presigned URL on S3.
4. Confirm Parts: After successfully uploading a part to S3, the client confirms this with ValetUpload. The service tracks confirmed parts in Redis.
5. Automatic Completion: Once all parts are confirmed, ValetUpload automatically completes the multipart upload on S3 and cleans up its internal tracking state.

This approach offloads the S3-specific logic from the client, enhancing security and simplifying client implementations.

Features

• Multipart Upload Orchestration: Manages the entire lifecycle of S3 multipart uploads, from initiation to completion.
• Secure Presigned URLs: Generates temporary, single-use presigned URLs for clients to upload file parts directly to S3, eliminating the need for client-side AWS credentials.
• Ephemeral State Tracking: Utilizes Redis to keep track of upload metadata and confirmed parts with a sliding expiration window, ensuring transient state doesn't persist longer than necessary.
• Automatic Upload Completion: Detects when all parts of an upload have been confirmed and automatically finalizes the multipart upload in S3.
• Error Handling: Provides clear API responses for common issues like unknown uploads or invalid requests.

Start a New Multipart Upload

Clients can initiate a new upload, receiving an uploadId to reference future operations and an s3UploadId for the underlying S3 process.

sequenceDiagram
    actor Client
    participant API as "ValetUpload API"
    participant S3 as "AWS S3"
    participant Redis as "Redis PartTracker"

    Client->>API: POST /uploads (filename, totalParts)
    API->>S3: Start Multipart Upload (filename)
    S3-->>API: s3UploadId
    API->>Redis: createUpload(uploadId, s3UploadId, filename, totalParts, ttl)
    Redis-->>API: OK
    API-->>Client: 200 OK (upload_id, s3_upload_id)

Loading

Request a Presigned URL for a Part

For each part of the file, the client requests a unique presigned URL. This URL is valid for a limited time and allows direct upload to S3.

sequenceDiagram
    actor Client
    participant API as "ValetUpload API"
    participant Redis as "Redis PartTracker"
    participant S3Presigner as "AWS S3 Presigner"

    Client->>API: POST /uploads/{uploadId}/parts/{partNumber}/url
    API->>Redis: getUploadMeta(uploadId)
    Redis-->>API: UploadMeta (s3UploadId, objectKey, totalParts)
    alt Upload Not Found
        Redis-->>API: Empty
        API-->>Client: 404 Not Found
    else Upload Exists
        API->>S3Presigner: presignPartUrl(s3UploadId, objectKey, partNumber, expiresIn)
        S3Presigner-->>API: presignedUrl
        API-->>Client: 200 OK (presigned_url, expires_in)
    end

Loading

Confirm a Part and Complete the Upload

After successfully uploading a part to S3, the client informs ValetUpload. The service tracks this, and once all parts are confirmed, it finalizes the S3 multipart upload.

sequenceDiagram
    actor Client
    participant API as "ValetUpload API"
    participant Redis as "Redis PartTracker"
    participant S3 as "AWS S3"

    Client->>API: POST /uploads/{uploadId}/parts/{partNumber}/confirm
    API->>Redis: getUploadMeta(uploadId)
    Redis-->>API: UploadMeta (s3UploadId, objectKey, totalParts)
    alt Upload Not Found
        Redis-->>API: Empty
        API-->>Client: 404 Not Found
    else Upload Exists
        API->>Redis: confirmPart(uploadId, partNumber, ttl)
        Redis-->>API: OK
        API->>Redis: countConfirmedParts(uploadId)
        Redis-->>API: confirmedPartsCount
        alt All parts confirmed (totalParts == confirmedPartsCount)
            API->>S3: completeUpload(s3UploadId, objectKey)
            S3-->>API: OK
            API->>Redis: deleteUpload(uploadId)
            Redis-->>API: OK
        end
        API-->>Client: 200 OK
    end

Loading

System Architecture / Design

The ValetUpload service is designed as a lightweight, stateless (aside from Redis for tracking) Java microservice. It acts as an intermediary between clients and AWS S3, using Redis as an ephemeral state store for ongoing multipart uploads.

flowchart LR
    Client["Client (Web/Mobile App)"]
    API["ValetUpload API (Javalin)"]
    S3[("AWS S3 Blob Storage")]
    Redis[("Redis Part Tracker")]

    Client -- "REST API Calls" --> API
    API -- "Starts MP Upload, Gets Presigned URLs, Completes MP Upload" --> S3
    API -- "Stores/Retrieves Upload Metadata, Confirmed Parts" --> Redis

Loading

API Documentation

The ValetUpload API provides three main endpoints to manage the multipart upload process. All responses are JSON.

POST /uploads

Description: Initiates a new multipart upload. The service generates a unique upload_id for tracking and an s3_upload_id for the underlying S3 operation.

Request:

{
  "filename": "my-large-file.zip",
  "total_parts": 10
}

Response:

{
  "upload_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "s3_upload_id": "ABCDEFG1234567890HIJKLMNOPQRSTUVW"
}

Errors:

• 500: Internal server error (e.g., S3 or Redis issues)

POST /uploads/{uploadId}/parts/{partNumber}/url

Description: Requests a presigned URL for a specific part of an ongoing upload. The partNumber is 1-indexed. The URL allows the client to upload the file part directly to S3.

Request: (No request body needed)

Response:

{
  "presigned_url": "https://your-s3-bucket.s3.amazonaws.com/my-large-file.zip?partNumber=1&uploadId=ABCDEFG1234...&X-Amz-Signature=...",
  "expires_in": 900
}

Errors:

• 400: Invalid part number (if partNumber is not an integer)
• 404: Upload not found (if uploadId is invalid or expired)
• 500: Internal server error (e.g., S3 presigning issues)

POST /uploads/{uploadId}/parts/{partNumber}/confirm

Description: Confirms that a specific file part has been successfully uploaded to S3. If all parts are confirmed, the service will automatically complete the multipart upload in S3 and clean up its internal state.

Request: (No request body needed)

Response: (200 OK with an empty body upon successful confirmation)

Errors:

• 400: Invalid part number (if partNumber is not an integer)
• 404: Upload not found (if uploadId is invalid or expired)
• 500: Internal server error (e.g., Redis issues or S3 completion failure)

Technologies Used

Technology	Description
Java 21	The core programming language.
Javalin 6.7.0	A lightweight web framework for building the REST API.
AWS S3 SDK	Java SDK for interacting with Amazon S3 for blob storage.
Jedis 7.4.1	A robust client for Redis, used for ephemeral part tracking.
Jackson	For JSON serialization and deserialization in the API.
Maven	Build automation tool for Java projects.

License

This project is licensed under the MIT License. See the repository for details.

Author Info

Connect with me!

• LinkedIn: Daniel Popoola
• X (Twitter): @iamuchihadan

---