diff --git a/.optimize-cache.json b/.optimize-cache.json index c13f849465a..099d2b517dc 100644 --- a/.optimize-cache.json +++ b/.optimize-cache.json @@ -591,6 +591,7 @@ "static/images/blog/everything-new-with-appwrite-1.5/1.5-recap.png": "1d3c646f6902757152d98861630c1952631a54f222af7f8476f53f4d0d3c59f2", "static/images/blog/everything-new-with-appwrite-1.5/messaging-console.png": "769b7df74c9107a5ccacfe87722293adbfbd91ab702c79b03838c2368e9971ac", "static/images/blog/examples-of-vibe-coding/cover.png": "745d0e65c7981fe852b2e1797c3163cd4e4c147227b906cf305019137cb4624f", + "static/images/blog/faster-storage-uploads-parallel-chunks/cover.png": "4565a9b19b4cafad3ed9ca3f2d4c6e4437379c5d10de6bad64dc1fb85590e49b", "static/images/blog/february-and-march-product-update-realtime-queries-appwrite-skills-and-new-database-features/Announcing_Appwrite_Skills__Give_your_AI_agents_Appwrite_expertise.png": "f6556f4786b55f53d06ca4c1a74ce0e488fa898099bf6458cab3e525f0a05d54", "static/images/blog/february-and-march-product-update-realtime-queries-appwrite-skills-and-new-database-features/Announcing_Realtime_Channel_helpers__Type-safe_subscriptions_made_simple.png": "a937f5b617fcbaa1d8d6af38f061f132a7590b85f93b104ee22119e80d5ed6d2", "static/images/blog/february-and-march-product-update-realtime-queries-appwrite-skills-and-new-database-features/comm_recoggg.png": "207e8acd544ebdd118f9aafb9d049dfa7fbb87868947d3df8b0fe86288848df0", diff --git a/src/lib/generated/github-stars.json b/src/lib/generated/github-stars.json index b3e078c531f..39a0e02f561 100644 --- a/src/lib/generated/github-stars.json +++ b/src/lib/generated/github-stars.json @@ -1,4 +1,4 @@ { - "stars": 55973, - "fetchedAt": "2026-05-04T21:03:24.879Z" + "stars": 55982, + "fetchedAt": "2026-05-05T10:41:28.857Z" } diff --git a/src/routes/blog/post/faster-storage-uploads-parallel-chunks/+page.markdoc b/src/routes/blog/post/faster-storage-uploads-parallel-chunks/+page.markdoc new file mode 100644 index 00000000000..774256d0333 --- /dev/null +++ b/src/routes/blog/post/faster-storage-uploads-parallel-chunks/+page.markdoc @@ -0,0 +1,363 @@ +--- +layout: post +title: "Up to 7x faster Appwrite Storage uploads with parallel chunks" +description: Appwrite SDKs now upload Storage chunks in parallel on runtimes with native concurrency. Our Node SDK benchmarks show up to 7.10x faster uploads on large files, with no API changes. +date: 2026-05-18 +cover: /images/blog/faster-storage-uploads-parallel-chunks/cover.avif +timeToRead: 9 +author: eldad-fux +category: announcement +featured: false +--- + +Uploading large files to [Appwrite Storage](/docs/products/storage) should feel snappy on a good connection and **bearable on real-world networks**. Until now, SDKs typically sent file chunks one after another. That approach is simple and reliable, but it leaves performance on the table. Each chunk waits for the previous one, so you rarely saturate bandwidth or use the browser's ability to run several HTTP requests in parallel. + +We are releasing updated **Appwrite SDKs** that upload **multiple chunks at the same time**, with defaults tuned for how browsers and HTTP clients actually behave. The speedup scales with file size: in our Node SDK benchmarks, a 1.28 GB upload dropped from **4 minutes 44 seconds to under 40 seconds, a 7.10x improvement** at the default concurrency of 8. + +# Why sequential chunks cap your speed + +Chunked uploads exist so large files do not need to live entirely in memory and so failures can be retried at chunk granularity. The tradeoff is that strictly sequential chunking means: + +- **Underused bandwidth** - the connection often sits idle while the client waits on the next chunk. +- **Latency stacked in series** - every chunk pays its own round trip one after another. +- **Browser limits left unused** - browsers allow multiple connections per origin, but a single in-flight upload does not use that capacity. + +The server must accept chunks in a well-defined way and assemble a complete file safely. Making uploads parallel required both smarter clients and a backend that could handle out-of-order and concurrent chunk writes without corrupting metadata or final assembly. + +# What we shipped + +The same establish-then-parallelize pattern is implemented everywhere the host runtime supports it. Each SDK uses that language's native concurrency primitives so multiple chunk requests run together without blocking each other on the wire. The updated SDKs: + +- Establish the upload by sending the first chunk in a controlled way to obtain an upload identifier. +- Upload the remaining chunks concurrently up to a fixed maximum parallelism. + +The examples below are the same `createFile` calls you use today. Point them at a large file, bump the SDK and server, and uploads get faster with no extra parameters for parallelism. + +{% multicode %} +```client-web +import { Client, Storage, ID } from 'appwrite'; + +const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + +const storage = new Storage(client); + +const uploaded = await storage.createFile({ + bucketId: 'videos', + fileId: ID.unique(), + file: document.getElementById('uploader').files[0] +}); +``` +```client-flutter +import 'package:appwrite/appwrite.dart'; + +final client = Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + +final storage = Storage(client); + +final uploaded = await storage.createFile( + bucketId: 'videos', + fileId: ID.unique(), + file: InputFile.fromPath( + path: './large-video.mp4', + filename: 'large-video.mp4', + ), +); +``` +```client-react-native +import { Client, Storage, ID } from 'react-native-appwrite'; + +const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject(''); + +const storage = new Storage(client); + +const uploaded = await storage.createFile({ + bucketId: 'videos', + fileId: ID.unique(), + file: { + name: 'large-video.mp4', + type: 'video/mp4', + size: fileSize, + uri: fileUri + } +}); +``` +```client-apple +import Appwrite + +let client = Client() + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + +let storage = Storage(client) + +let uploaded = try await storage.createFile( + bucketId: "videos", + fileId: ID.unique(), + file: InputFile.fromPath("./large-video.mp4") +) +``` +```client-android-kotlin +import io.appwrite.Client +import io.appwrite.ID +import io.appwrite.models.InputFile +import io.appwrite.services.Storage + +suspend fun upload() { + val client = Client(applicationContext) + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + + val storage = Storage(client) + + val uploaded = storage.createFile( + bucketId = "videos", + fileId = ID.unique(), + file = InputFile.fromPath("./large-video.mp4") + ) +} +``` +```server-nodejs +import { Client, Storage, ID } from 'node-appwrite'; +import { InputFile } from 'node-appwrite/file'; + +const client = new Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject('') + .setKey(''); + +const storage = new Storage(client); + +const uploaded = await storage.createFile({ + bucketId: 'videos', + fileId: ID.unique(), + file: InputFile.fromPath('./large-video.mp4', 'large-video.mp4') +}); +``` +```server-python +from appwrite.client import Client +from appwrite.services.storage import Storage +from appwrite.id import ID +from appwrite.input_file import InputFile + +client = Client() +client.set_endpoint('https://.cloud.appwrite.io/v1') +client.set_project('') +client.set_key('') + +storage = Storage(client) + +uploaded = storage.create_file( + bucket_id='videos', + file_id=ID.unique(), + file=InputFile.from_path('./large-video.mp4'), +) +``` +```server-dart +import 'package:dart_appwrite/dart_appwrite.dart'; + +final client = Client() + .setEndpoint('https://.cloud.appwrite.io/v1') + .setProject('') + .setKey(''); + +final storage = Storage(client); + +final uploaded = await storage.createFile( + bucketId: 'videos', + fileId: ID.unique(), + file: InputFile.fromPath( + path: './large-video.mp4', + filename: 'large-video.mp4', + ), +); +``` +```server-php +setEndpoint('https://.cloud.appwrite.io/v1') + ->setProject('') + ->setKey(''); + +$storage = new Storage($client); + +$uploaded = $storage->createFile( + bucketId: 'videos', + fileId: ID::unique(), + file: InputFile::withPath('./large-video.mp4'), +); +``` +```server-ruby +require 'appwrite' + +include Appwrite + +client = Client.new + .set_endpoint('https://.cloud.appwrite.io/v1') + .set_project('') + .set_key('') + +storage = Storage.new(client) + +uploaded = storage.create_file( + bucket_id: 'videos', + file_id: ID.unique, + file: InputFile.from_path('./large-video.mp4'), +) +``` +```server-go +package main + +import ( + "github.com/appwrite/sdk-for-go/client" + "github.com/appwrite/sdk-for-go/file" + "github.com/appwrite/sdk-for-go/id" + "github.com/appwrite/sdk-for-go/storage" +) + +func main() { + clt := client.New( + client.WithEndpoint("https://.cloud.appwrite.io/v1"), + client.WithProject(""), + client.WithKey(""), + ) + + service := storage.New(clt) + + uploaded, err := service.CreateFile( + "videos", + id.Unique(), + file.NewInputFile("./large-video.mp4", "large-video.mp4"), + ) + _ = uploaded + _ = err +} +``` +```server-rust +use appwrite::Client; +use appwrite::services::Storage; +use appwrite::InputFile; +use appwrite::id::ID; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let client = Client::new() + .set_endpoint("https://.cloud.appwrite.io/v1") + .set_project("") + .set_key(""); + + let storage = Storage::new(&client); + + let file = InputFile::from_path("./large-video.mp4", None).await?; + + let uploaded = storage.create_file( + "videos", + ID::unique(), + file, + None, + ).await?; + + let _ = uploaded; + Ok(()) +} +``` +```server-kotlin +import io.appwrite.Client +import io.appwrite.ID +import io.appwrite.models.InputFile +import io.appwrite.services.Storage + +suspend fun upload() { + val client = Client() + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + .setKey("") + + val storage = Storage(client) + + val uploaded = storage.createFile( + bucketId = "videos", + fileId = ID.unique(), + file = InputFile.fromPath("./large-video.mp4") + ) +} +``` +```server-swift +import Appwrite + +let client = Client() + .setEndpoint("https://.cloud.appwrite.io/v1") + .setProject("") + .setKey("") + +let storage = Storage(client) + +let uploaded = try await storage.createFile( + bucketId: "videos", + fileId: ID.unique(), + file: InputFile.fromPath("./large-video.mp4") +) +``` +```server-dotnet +using Appwrite; +using Appwrite.Models; +using Appwrite.Services; + +Client client = new Client() + .SetEndPoint("https://.cloud.appwrite.io/v1") + .SetProject("") + .SetKey(""); + +Storage storage = new Storage(client); + +File uploaded = await storage.CreateFile( + bucketId: "videos", + fileId: ID.Unique(), + file: InputFile.FromPath("./large-video.mp4") +); +``` +{% /multicode %} + +# Benchmarks + +We benchmarked the Node SDK uploading files from 10 MB up to 1.28 GB, comparing the old sequential client against the new parallel client at the default concurrency of 8. + +| File size | Sequential | Parallel | Speedup | Concurrency | Chunks | +| --- | --- | --- | --- | --- | --- | +| 10 MB | 2,650 ms | 2,472 ms | 1.07x | 1 | 2 | +| 20 MB | 4,589 ms | 2,434 ms | 1.89x | 3 | 4 | +| 40 MB | 9,413 ms | 2,680 ms | 3.51x | 7 | 8 | +| 80 MB | 18,233 ms | 4,037 ms | 4.52x | 8 | 16 | +| 160 MB | 36,606 ms | 6,545 ms | 5.59x | 8 | 32 | +| 320 MB | 71,036 ms | 11,451 ms | 6.20x | 8 | 64 | +| 640 MB | 141,923 ms | 20,863 ms | 6.80x | 8 | 128 | +| 1.28 GB | 283,823 ms | 39,956 ms | **7.10x** | 8 | 256 | + +The pattern is clear: + +- **Small files** (a single chunk or two) cannot benefit from concurrency and stay close to the sequential baseline. +- **Large files** have enough chunks to saturate the worker pool, so the speedup climbs steeply, reaching up to 7.10x at 1.28 GB. + +Your mileage will vary with region, device, bucket location, and network, but the larger the file, the larger the win. + +# Get started + +Parallel chunked uploads are now available on Appwrite Cloud. Upgrade your Appwrite SDK to the latest release for your language and your existing `createFile` calls inherit the faster uploads automatically - no API migration, no extra configuration. + +The Appwrite Console also uses parallel chunking, so file uploads through the Console are faster too. + +# More resources + +- [Appwrite Storage documentation](/docs/products/storage) +- [The easiest way to add file uploads to your app](/blog/post/easiest-file-uploads) diff --git a/src/routes/changelog/(entries)/2026-05-18-1.markdoc b/src/routes/changelog/(entries)/2026-05-18-1.markdoc new file mode 100644 index 00000000000..2670a86745b --- /dev/null +++ b/src/routes/changelog/(entries)/2026-05-18-1.markdoc @@ -0,0 +1,17 @@ +--- +layout: changelog +title: "Up to 7x faster Appwrite Storage uploads with parallel chunks" +description: Appwrite SDKs now upload Storage chunks in parallel on runtimes with native concurrency. Our Node SDK benchmarks show up to 7.10x faster uploads on large files, with no API changes for developers. +date: 2026-05-18 +cover: /images/blog/faster-storage-uploads-parallel-chunks/cover.avif +--- + +Appwrite SDKs now upload **Storage** file chunks **in parallel** where the host runtime supports overlapping HTTP requests. Chunking, concurrency limits, and ordering are handled **inside the client**; your **`createFile`** calls stay the same. + +In our Node SDK benchmarks, a 1.28 GB upload dropped from 4 minutes 44 seconds to under 40 seconds, up to a **7.10x** improvement at the default concurrency of 8. Smaller files see proportionally smaller gains since they have fewer chunks to overlap. + +Available on Appwrite Cloud today. + +{% arrow_link href="/blog/post/faster-storage-uploads-parallel-chunks" %} +Read the announcement +{% /arrow_link %} diff --git a/src/routes/docs/apis/rest/+page.markdoc b/src/routes/docs/apis/rest/+page.markdoc index a1f19a42ee4..89e38712406 100644 --- a/src/routes/docs/apis/rest/+page.markdoc +++ b/src/routes/docs/apis/rest/+page.markdoc @@ -151,9 +151,9 @@ Learn more in the [user impersonation docs](/docs/products/auth/impersonation). # Files {% #files %} -Appwrite implements resumable, chunked uploads for files larger than 5MB. Chunked uploads send files in chunks of 5MB to reduce memory footprint and increase resilience when handling large files. [Appwrite SDKs](/docs/sdks) will automatically handle chunked uploads, but it is possible to implement this with the REST API directly. +Appwrite implements resumable, chunked uploads for files larger than 5MB. Each chunk is up to **5MB**, which reduces memory footprint and increases resilience when handling large files. [Appwrite SDKs](/docs/sdks) split uploads, attach the headers below, and on runtimes with native concurrency they may also **send multiple chunk requests in parallel** for higher throughput, while your high-level upload code stays unchanged. You can still implement chunked uploads with the REST API directly. -Upload endpoints in Appwrite, such as [Create File](/docs/references/cloud/client-web/storage#createFile) and [Create Deployment](/docs/references/cloud/server-nodejs/functions#createDeployment), are different from other endpoints. These endpoints take multipart form data instead of JSON data. To implement chunked uploads, you will need to implement the following headers. If you wish, this logic is already available in any of the [Appwrite SDKs](/docs/sdks). +Upload endpoints in Appwrite, such as [Create File](/docs/references/cloud/client-web/storage#createFile) and [Create Deployment](/docs/references/cloud/server-nodejs/functions#createDeployment), are different from other endpoints. These endpoints take multipart form data instead of JSON data. To implement chunked uploads over REST, send **one HTTP request per chunk** using the headers in the tables below. The **first** request establishes the file; every later chunk must repeat the same file **id** in `X-Appwrite-ID` and set `Content-Range` to the byte span carried in that request. After the first response, you may **issue the remaining chunk requests sequentially or in parallel** (for example from a thread pool or async tasks), as long as each byte of the file is uploaded **exactly once** and each part stays within the maximum chunk size. For Storage-specific guidance (including SDK behavior), see [Upload and download](/docs/products/storage/upload-download#large-files). {% table %} diff --git a/src/routes/docs/references/[version]/[platform]/[service]/descriptions/storage.md b/src/routes/docs/references/[version]/[platform]/[service]/descriptions/storage.md index 846afcedf77..18bf27abff1 100644 --- a/src/routes/docs/references/[version]/[platform]/[service]/descriptions/storage.md +++ b/src/routes/docs/references/[version]/[platform]/[service]/descriptions/storage.md @@ -1,5 +1,7 @@ The Storage service allows you to manage your project files. Using the Storage service, you can upload, view, download, and query all your project files. +Large files are uploaded in 5MB chunks. Appwrite SDKs handle chunking for you and, on runtimes with native concurrency, can upload multiple chunks in parallel for faster throughput without changing your `createFile`-style calls. + Files are managed using buckets. Storage buckets are similar to Tables we have in our [Databases](/docs/products/databases) service. The difference is, buckets also provide more power to decide what kinds of files, what sizes you want to allow in that bucket, whether or not to encrypt the files, scan with antivirus and more. Using Appwrite permissions architecture, you can assign read or write access to each bucket or file in your project for either a specific user, team, user role, or even grant it with public access (`any`). You can learn more about [how Appwrite handles permissions and access control](/docs/advanced/platform/permissions). diff --git a/src/routes/docs/tooling/command-line/buckets/+page.markdoc b/src/routes/docs/tooling/command-line/buckets/+page.markdoc index 9cb9337d5c1..f09a131d0e4 100644 --- a/src/routes/docs/tooling/command-line/buckets/+page.markdoc +++ b/src/routes/docs/tooling/command-line/buckets/+page.markdoc @@ -101,7 +101,7 @@ appwrite storage [COMMAND] [OPTIONS] * Get a list of all the user files. You can use the query params to filter your results. --- * `create-file [options]` -* Create a new file. Before using this route, you should create a new bucket resource using either a [server integration](https://appwrite.io/docs/server/storage#storageCreateBucket) API or directly from your Appwrite console. Larger files should be uploaded using multiple requests with the [content-range](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range) header to send a partial request with a maximum supported chunk of '5MB'. The 'content-range' header values should always be in bytes. When the first request is sent, the server will return the **File** object, and the subsequent part request must include the file's **id** in 'x-appwrite-id' header to allow the server to know that the partial upload is for the existing file and not for a new one. If you're creating a new file using one of the Appwrite SDKs, all the chunking logic will be managed by the SDK internally. +* Create a new file. Before using this route, you should create a new bucket resource using either a [server integration](https://appwrite.io/docs/server/storage#storageCreateBucket) API or directly from your Appwrite console. Larger files should be uploaded using multiple requests with the [content-range](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range) header to send a partial request with a maximum supported chunk of '5MB'. The 'content-range' header values should always be in bytes. When the first request is sent, the server will return the **File** object, and the subsequent part request must include the file's **id** in 'x-appwrite-id' header to allow the server to know that the partial upload is for the existing file and not for a new one. If you're creating a new file using one of the Appwrite SDKs, all the chunking logic will be managed by the SDK internally, including sending multiple chunk requests in parallel when the runtime supports it. --- * `get-file [options]` * Get a file by its unique ID. This endpoint response returns a JSON object with the file metadata. diff --git a/static/images/blog/faster-storage-uploads-parallel-chunks/cover.avif b/static/images/blog/faster-storage-uploads-parallel-chunks/cover.avif new file mode 100644 index 00000000000..d6510845f6e Binary files /dev/null and b/static/images/blog/faster-storage-uploads-parallel-chunks/cover.avif differ