Handling Large File Uploads in Go Without Breaking the Server
RA
Rafly Aziz Abdillah
47 menit baca
Chunked File Upload in Go: A Practical Guide
There is a certain kind of pain that every backend developer eventually encounters. You build a file upload endpoint, everything works fine in local testing with a 500 KB sample file, and then someone in production tries to upload a 2 GB video. The request times out. The user retries. The timeout happens again. Someone files a bug report with a subject line that contains the word "urgent." You stare at the ceiling.
This article is about solving that problem properly, using chunked file upload.
We will walk through a full working implementation in Go, using the Fiber web framework, GORM, and a clean layered architecture. The code shown here is the same code in the project repository, so you can run it directly without adapting anything.
The Core Problem with Naive File Uploads
A conventional file upload works like this: the client opens a connection, sends the entire file body in one HTTP request, and waits for the server to respond. This is fine for files under a few megabytes. Beyond that, you start running into real problems.
Connection timeouts. HTTP servers have request timeouts. Proxies like nginx have their own. A slow upload on a mobile connection can easily exceed 30 to 60 seconds, which is where many default configurations sit.
Memory pressure. Depending on how the server reads the request body, the entire file can land in memory at once. If ten users upload simultaneously, that multiplies. The server starts struggling.
No resume capability. If the upload fails at 95%, the user has to start over from zero. This is unacceptable for large files and genuinely infuriating for users.
No progress reporting. A single opaque request gives the client no way to tell the user how far along the upload is.
Chunked upload solves all of these by breaking the file into smaller pieces before sending them. Each piece is a separate, short HTTP request. The server stores each piece temporarily, and when all pieces have arrived, it assembles them into the original file.
How It Works: The Flow
Here's the full picture — client side, server side, and yes, the retry path is included because that's actually where chunked upload earns its keep.
| Step | Client | Server |
|---|---|---|
| 1. Initiate session | POST /api/v1/files/initiate<br>{ original_name, total_chunks, ... } | Create DB record and return file_id.<br>{ id, upload_status: "pending", ... } |
| 2. Upload chunk 0 | POST /api/v1/files/:id/chunks<br>chunk_index=0, chunk=<bytes> | Write chunk_00000 to disk.<br>{ uploaded_chunks: 1, total_chunks: 5 } |
| 3. Upload chunk 1 | POST /api/v1/files/:id/chunks<br>chunk_index=1, chunk=<bytes> | Write chunk_00001 to disk.<br>{ uploaded_chunks: 2, total_chunks: 5 } |
| ... | Repeat for each chunk index | Repeat: store chunk and update progress |
| N. Upload final chunk | POST /api/v1/files/:id/chunks<br>chunk_index=4, chunk=<bytes> | Write chunk_00004 to disk, detect uploaded == total, assemble chunk_00000 + ... + chunk_00004, move to ./storage/, update DB to status="completed" with file_url, then return { upload_status: "completed", file_url: "..." } |
The client splits. The server receives and assembles. And if something fails in transit, only that chunk gets retried — not the whole file. That's the whole point.
Here's what happens on the server side at each step:
Step 1 — initiate: Creates a files record with upload_status = "pending" and uploaded_chunks = 0. Returns the id that identifies the session for all subsequent chunk requests.
Step 2 — upload chunks: Each chunk is a separate multipart request. The server writes it to a temp directory as chunk_00000, chunk_00001, etc. (zero-padded for deterministic ordering). The counter only increments if this chunk index is new — so duplicate sends due to retries are safe and do not corrupt the count.
Step 3 — last chunk: When uploaded_chunks == total_chunks, the server concatenates all chunks in index order, moves the assembled file to ./storage/, updates the database record to completed, and starts a background goroutine to clean up the temp directory. The HTTP response goes out before the cleanup finishes.
Project Structure
Before we get into the code, here's how things are laid out.
text
chunk-tutorial/
├── cmd/
│ └── main.go ← Entry point, manual dependency injection
├── config/
│ └── config.go ← Env-variable loading
└── internal/
├── domain/file/
│ ├── entity.go ← File entity (single DB table)
│ ├── repository.go ← Repository interface
│ └── errors.go ← Domain-level sentinel errors
├── application/file/
│ ├── dto.go ← Input DTOs
│ ├── storage.go ← StorageProvider port interface
│ ├── service.go ← Service interface
│ └── service_impl.go ← Business logic
├── infrastructure/
│ ├── database/postgres.go ← GORM connection + AutoMigrate
│ ├── localstorage/storage.go ← Local disk storage adapter
│ └── repository/file_repository.go ← GORM implementation
└── interfaces/http/
├── handler/file_handler.go ← HTTP handlers
└── router/router.go ← Route registrationThe dependency rule is simple: inner layers know nothing about outer layers. The domain knows nothing about HTTP or GORM. The application layer knows about the domain but not about Fiber or PostgreSQL. Infrastructure adapts external tools to fit what the application layer needs.
The Domain Layer
Everything starts with the entity — and there's only one table in this whole project.
go
// internal/domain/file/entity.go
package file
import (
"time"
"github.com/google/uuid"
"gorm.io/gorm"
)
// UploadStatus represents the current state of a file upload session.
type UploadStatus string
const (
StatusPending UploadStatus = "pending"
StatusCompleted UploadStatus = "completed"
StatusFailed UploadStatus = "failed"
)
// File is the core domain entity representing an uploaded file.
type File struct {
ID uuid.UUID `gorm:"type:uuid;primaryKey" json:"id"`
OriginalName string `gorm:"column:original_name;not null" json:"original_name"`
FileName string `gorm:"column:file_name;not null" json:"file_name"`
MimeType string `gorm:"column:mime_type" json:"mime_type"`
FileSize int64 `gorm:"column:file_size" json:"file_size"`
StoragePath string `gorm:"column:storage_path" json:"storage_path,omitempty"`
FileURL string `gorm:"column:file_url" json:"file_url,omitempty"`
UploadStatus UploadStatus `gorm:"column:upload_status;default:'pending'" json:"upload_status"`
TotalChunks int `gorm:"column:total_chunks" json:"total_chunks"`
UploadedChunks int `gorm:"column:uploaded_chunks;default:0" json:"uploaded_chunks"`
CreatedAt time.Time `gorm:"column:created_at;autoCreateTime" json:"created_at"`
UpdatedAt time.Time `gorm:"column:updated_at;autoUpdateTime" json:"updated_at"`
}
func (File) TableName() string {
return "files"
}
// BeforeCreate generates a new UUID for the file before it is inserted.
func (f *File) BeforeCreate(_ *gorm.DB) error {
if f.ID == uuid.Nil {
f.ID = uuid.New()
}
return nil
}Two fields deserve attention. StoragePath is the absolute path on disk where the assembled file lives after all chunks arrive. FileURL is the public-facing download URL that the API returns to clients. They serve different purposes: one is an internal implementation detail, the other is what you expose.
TotalChunks and UploadedChunks together give you progress at any moment: float64(UploadedChunks) / float64(TotalChunks) * 100.
The repository interface is defined in the domain layer so that nothing in the domain depends on GORM:
go
// internal/domain/file/repository.go
package file
import "github.com/google/uuid"
type Repository interface {
Create(file *File) error
FindByID(id uuid.UUID) (*File, error)
FindAll() ([]*File, error)
Update(file *File) error
Delete(id uuid.UUID) error
}And sentinel errors, because returning plain strings from a service layer is how you end up with inconsistent HTTP status codes across every handler in the codebase:
go
// internal/domain/file/errors.go
package file
import "errors"
var (
ErrNotFound = errors.New("file not found")
ErrAlreadyComplete = errors.New("file upload already completed")
ErrChunkOutOfRange = errors.New("chunk index out of range")
)Handlers can call errors.Is(err, file.ErrNotFound) and return 404 without parsing error strings. This is the right way.
The Application Layer
The Storage Port
Here's the key design decision: the application layer needs to store assembled files somewhere, but it shouldn't care where or how. That's what the StorageProvider interface is for:
go
// internal/application/file/storage.go
package fileapp
import "github.com/google/uuid"
// StorageProvider is the port the application layer uses to interact with file storage.
// Swap the adapter (local disk, S3, GCS, Cloudinary) without touching this interface.
type StorageProvider interface {
Store(assembledPath string, fileID uuid.UUID, originalName string) (storagePath string, fileURL string, err error)
Delete(storagePath string) error
}This is a critical design decision. Today the project stores files on local disk. If tomorrow you want to push files to AWS S3 or ImageKit, you write a new adapter that satisfies this interface and swap it in main.go. Nothing else changes. The service does not know or care.
The Service
The service is where the actual chunk upload logic lives. The interesting part is UploadChunk:
go
// internal/application/file/service_impl.go (excerpt)
func (s *fileService) UploadChunk(req *UploadChunkRequest) (*domainfile.File, error) {
f, err := s.repo.FindByID(req.FileID)
if err != nil {
return nil, err
}
if f.UploadStatus == domainfile.StatusCompleted {
return nil, domainfile.ErrAlreadyComplete
}
if req.ChunkIndex < 0 || req.ChunkIndex >= f.TotalChunks {
return nil, domainfile.ErrChunkOutOfRange
}
chunkPath := s.chunkPath(req.FileID, req.ChunkIndex)
// Idempotency: only increment counter when this chunk arrives for the first time.
isNewChunk := !fileExists(chunkPath)
if err := os.WriteFile(chunkPath, req.ChunkData, 0600); err != nil {
return nil, fmt.Errorf("write chunk %d: %w", req.ChunkIndex, err)
}
if isNewChunk {
f.UploadedChunks++
}
// When all chunks have arrived, assemble and move to permanent storage.
if f.UploadedChunks >= f.TotalChunks {
assembled := filepath.Join(s.tempDir, f.ID.String()+"_assembled")
if err := s.assemble(s.chunkDir(req.FileID), assembled, f.TotalChunks); err != nil {
s.markFailed(f)
return nil, fmt.Errorf("assemble chunks: %w", err)
}
storagePath, fileURL, err := s.storage.Store(assembled, req.FileID, f.OriginalName)
if err != nil {
s.markFailed(f)
os.Remove(assembled)
os.RemoveAll(s.chunkDir(req.FileID))
return nil, fmt.Errorf("move to storage: %w", err)
}
f.StoragePath = storagePath
f.FileURL = fileURL
f.UploadStatus = domainfile.StatusCompleted
// Temp cleanup runs after the response is returned.
go func() {
os.Remove(assembled)
os.RemoveAll(s.chunkDir(req.FileID))
}()
}
if err := s.repo.Update(f); err != nil {
return nil, fmt.Errorf("update file record: %w", err)
}
return f, nil
}A few things worth calling out:
Idempotent re-uploads. The check isNewChunk := !fileExists(chunkPath) means if a client sends the same chunk index twice (which can happen on retries after a network error), the counter only increments once. The file is simply overwritten. This is correct behavior.
Assembly only when all chunks are present. The condition f.UploadedChunks >= f.TotalChunks triggers the assembly step. The assemble method reads chunks in order using zero-padded filenames (chunk_00000, chunk_00001, etc.) and concatenates them into a single file.
Background cleanup. Temp files are removed in a goroutine after s.repo.Update succeeds. The HTTP response goes out before cleanup finishes, which keeps latency lower for the last chunk.
Failure handling. If assembly or storage fails, markFailed sets the upload status to failed and persists it. The client can see this in a subsequent GET /api/v1/files/:id and decide whether to retry the whole upload.
The assembly itself is straightforward:
go
func (s *fileService) assemble(chunkDir, destPath string, totalChunks int) error {
dest, err := os.Create(destPath)
if err != nil {
return fmt.Errorf("create assembled file: %w", err)
}
defer dest.Close()
for i := 0; i < totalChunks; i++ {
chunk, err := os.ReadFile(filepath.Join(chunkDir, chunkFileName(i)))
if err != nil {
return fmt.Errorf("read chunk %d: %w", i, err)
}
if _, err := dest.Write(chunk); err != nil {
return fmt.Errorf("write chunk %d to assembled file: %w", i, err)
}
}
return nil
}
// chunkFileName returns a zero-padded filename so directory listings are ordered correctly.
func chunkFileName(index int) string {
return fmt.Sprintf("chunk_%05d", index)
}Zero-padding is important here. Without it, chunk_10 would sort before chunk_2 in a lexicographic listing, and your assembled file would be scrambled. With five digits (chunk_00010), you can handle up to 99,999 chunks before the sorting breaks, which covers any reasonable use case.
The Infrastructure Layer
Local Storage Adapter
The local storage adapter is a simple implementation of StorageProvider:
go
// internal/infrastructure/localstorage/storage.go
package localstorage
import (
"fmt"
"io"
"os"
"path/filepath"
"strings"
"github.com/google/uuid"
)
type LocalStorageAdapter struct {
storageDir string
baseURL string
}
func New(storageDir, baseURL string) (*LocalStorageAdapter, error) {
if err := os.MkdirAll(storageDir, 0750); err != nil {
return nil, fmt.Errorf("create storage directory: %w", err)
}
return &LocalStorageAdapter{
storageDir: storageDir,
baseURL: strings.TrimRight(baseURL, "/"),
}, nil
}
func (a *LocalStorageAdapter) Store(assembledPath string, fileID uuid.UUID, originalName string) (string, string, error) {
destName := fileID.String() + "_" + sanitize(originalName)
destPath := filepath.Join(a.storageDir, destName)
if err := moveFile(assembledPath, destPath); err != nil {
return "", "", fmt.Errorf("store file: %w", err)
}
fileURL := fmt.Sprintf("%s/api/v1/files/%s/download", a.baseURL, fileID.String())
return destPath, fileURL, nil
}
func (a *LocalStorageAdapter) Delete(storagePath string) error {
if err := os.Remove(storagePath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("delete file: %w", err)
}
return nil
}moveFile copies the assembled file to the permanent storage directory and removes the source. The reason it does not use os.Rename is that rename can return an error when crossing filesystem boundaries (for example, when your temp directory is on a different mount than your storage directory). An explicit copy-then-delete is more reliable.
The file gets named <uuid>_<original_name>, which avoids any collision between files with the same name uploaded by different users or at different times.
GORM Repository
The repository implementation is where GORM lives. Notice how it translates gorm.ErrRecordNotFound into the domain's ErrNotFound:
go
// internal/infrastructure/repository/file_repository.go (excerpt)
func (r *fileRepository) FindByID(id uuid.UUID) (*domainfile.File, error) {
var file domainfile.File
err := r.db.Where("id = ?", id).First(&file).Error
if err != nil {
if errors.Is(err, gorm.ErrRecordNotFound) {
return nil, domainfile.ErrNotFound
}
return nil, fmt.Errorf("repository find by id: %w", err)
}
return &file, nil
}The handler layer calls errors.Is(err, domainfile.ErrNotFound) and returns 404. It never has to know that GORM is involved. If you replaced GORM with sqlx tomorrow, this translation point is the only place you would change.
The Handler Layer
The HTTP handler for chunk uploads reads the multipart form, delegates to the service, and translates errors to HTTP status codes:
go
// internal/interfaces/http/handler/file_handler.go (excerpt)
func (h *FileHandler) UploadChunk(c *fiber.Ctx) error {
fileID, err := uuid.Parse(c.Params("id"))
if err != nil {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "invalid file id — must be a UUID",
})
}
chunkIndex, err := strconv.Atoi(c.FormValue("chunk_index"))
if err != nil || chunkIndex < 0 {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "chunk_index must be a non-negative integer",
})
}
chunkHeader, err := c.FormFile("chunk")
if err != nil {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "multipart field 'chunk' is required",
})
}
chunkFile, err := chunkHeader.Open()
if err != nil {
return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
"error": "failed to open uploaded chunk",
})
}
defer chunkFile.Close()
chunkData, err := io.ReadAll(chunkFile)
if err != nil {
return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
"error": "failed to read chunk data",
})
}
file, err := h.service.UploadChunk(&fileapp.UploadChunkRequest{
FileID: fileID,
ChunkIndex: chunkIndex,
ChunkData: chunkData,
})
if err != nil {
return h.mapServiceError(c, err)
}
msg := "chunk received"
if file.UploadStatus == domainfile.StatusCompleted {
msg = "all chunks received — file assembled and stored"
}
return c.JSON(fiber.Map{"message": msg, "data": file})
}Error mapping is centralized in one method:
go
func (h *FileHandler) mapServiceError(c *fiber.Ctx, err error) error {
switch {
case errors.Is(err, domainfile.ErrNotFound):
return c.Status(fiber.StatusNotFound).JSON(fiber.Map{"error": err.Error()})
case errors.Is(err, domainfile.ErrAlreadyComplete):
return c.Status(fiber.StatusConflict).JSON(fiber.Map{"error": err.Error()})
case errors.Is(err, domainfile.ErrChunkOutOfRange):
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{"error": err.Error()})
default:
return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{"error": err.Error()})
}
}Every handler calls this. You add error types to the domain once, and all handlers automatically handle them correctly.
Wiring It All Together
The main.go is the only place in the codebase where concrete types are instantiated and wired together:
go
// cmd/main.go
func main() {
cfg, err := config.Load()
if err != nil {
log.Fatalf("load config: %v", err)
}
db, err := database.Connect(cfg.DSN())
if err != nil {
log.Fatalf("connect database: %v", err)
}
storage, err := localstorage.New(cfg.StorageDir, cfg.AppURL)
if err != nil {
log.Fatalf("init local storage: %v", err)
}
fileRepo := repository.NewFileRepository(db)
fileService := fileapp.NewFileService(fileRepo, storage, cfg.TempDir)
fileHandler := handler.NewFileHandler(fileService)
app := fiber.New(fiber.Config{
BodyLimit: 50 * 1024 * 1024, // 50 MB per request
})
app.Use(recover.New())
app.Use(logger.New())
router.Register(app, fileHandler)
log.Printf("server listening on :%s", cfg.AppPort)
if err := app.Listen(":" + cfg.AppPort); err != nil {
log.Fatalf("server error: %v", err)
}
}This is manual dependency injection. No frameworks, no magic. You can read this top to bottom and understand exactly what depends on what. When storage is swapped for an S3 adapter, this is the only file that changes.
Trying It Yourself
Start the server with Air for live reload:
bash
cp .env.example .env
# Edit .env with your PostgreSQL credentials
airOr without Air:
bash
go run ./cmd/main.goNow test the full flow with curl. Split a local file into three parts first:
bash
# On Linux / macOS
split -b 1m photo.jpg chunk_
# This creates chunk_aa, chunk_ab, chunk_acThen run the upload sequence:
bash
# Step 1: Initiate the session
FILE_ID=$(curl -s -X POST http://localhost:3000/api/v1/files/initiate \
-H "Content-Type: application/json" \
-d '{"original_name":"photo.jpg","mime_type":"image/jpeg","total_chunks":3,"file_size":3145728}' \
| grep -o '"id":"[^"]*"' | cut -d'"' -f4)
echo "File ID: $FILE_ID"
# Step 2: Upload chunks (any order works)
curl -X POST http://localhost:3000/api/v1/files/$FILE_ID/chunks \
-F "chunk_index=0" -F "chunk=@chunk_aa"
curl -X POST http://localhost:3000/api/v1/files/$FILE_ID/chunks \
-F "chunk_index=1" -F "chunk=@chunk_ab"
curl -X POST http://localhost:3000/api/v1/files/$FILE_ID/chunks \
-F "chunk_index=2" -F "chunk=@chunk_ac"
# Step 3: Check the result
curl http://localhost:3000/api/v1/files/$FILE_ID
# Step 4: Download the assembled file
curl -OJ http://localhost:3000/api/v1/files/$FILE_ID/downloadAfter the third chunk, the response will include "upload_status": "completed" and a file_url. The assembled file will be in ./storage/.
The Client Side: Uploading from a Browser (React + TypeScript)
Everything above is the server's perspective. But the file has to get split somewhere first — and that's the client's job. Here's a complete implementation using React and TypeScript. No extra libraries needed beyond React itself.
Types first
typescript
// types/upload.ts
export interface InitiateResponse {
data: {
id: string;
};
}
export interface ChunkResponse {
data: {
uploaded_chunks: number;
total_chunks: number;
upload_status: "pending" | "completed" | "failed";
file_url: string;
};
}
export type UploadStatus =
| { kind: "idle" }
| { kind: "uploading"; progress: number }
| { kind: "done"; fileUrl: string }
| { kind: "error"; message: string };The upload logic (custom hook)
The upload logic lives in a custom hook so the component stays clean and the logic is independently testable.
typescript
// hooks/useChunkUpload.ts
import { useState, useCallback, useRef } from "react";
import type {
ChunkResponse,
InitiateResponse,
UploadStatus,
} from "../types/upload";
const API_BASE = "http://localhost:3000/api/v1";
const CHUNK_SIZE = 5 * 1024 * 1024; // 5 MB per chunk
const MAX_RETRY = 3;
async function uploadChunk(
fileID: string,
index: number,
blob: Blob,
fileName: string,
signal: AbortSignal,
): Promise<ChunkResponse> {
const form = new FormData();
form.append("chunk_index", String(index));
form.append("chunk", blob, fileName);
for (let attempt = 0; attempt < MAX_RETRY; attempt++) {
const res = await fetch(`${API_BASE}/files/${fileID}/chunks`, {
method: "POST",
body: form,
signal,
});
if (res.ok) return res.json() as Promise<ChunkResponse>;
// Last attempt — propagate as error
if (attempt === MAX_RETRY - 1) {
const text = await res.text();
throw new Error(`chunk ${index} failed (${res.status}): ${text}`);
}
// Linear backoff: 1 s, 2 s, ...
await new Promise((r) => setTimeout(r, 1000 * (attempt + 1)));
}
throw new Error(`chunk ${index}: unreachable`);
}
export function useChunkUpload() {
const [status, setStatus] = useState<UploadStatus>({ kind: "idle" });
const abortRef = useRef<AbortController | null>(null);
const upload = useCallback(async (file: File) => {
abortRef.current?.abort();
const controller = new AbortController();
abortRef.current = controller;
setStatus({ kind: "uploading", progress: 0 });
try {
const totalChunks = Math.ceil(file.size / CHUNK_SIZE);
// Step 1: register the session
const initRes = await fetch(`${API_BASE}/files/initiate`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
original_name: file.name,
mime_type: file.type || "application/octet-stream",
total_chunks: totalChunks,
file_size: file.size,
}),
signal: controller.signal,
});
if (!initRes.ok) throw new Error(`initiate failed: ${initRes.status}`);
const { data }: InitiateResponse = await initRes.json();
const fileID = data.id;
// Step 2: send chunks sequentially
for (let i = 0; i < totalChunks; i++) {
const start = i * CHUNK_SIZE;
const end = Math.min(start + CHUNK_SIZE, file.size);
const blob = file.slice(start, end); // no full copy — browser reads lazily
const { data: chunkData } = await uploadChunk(
fileID,
i,
blob,
file.name,
controller.signal,
);
const progress = Math.round(
(chunkData.uploaded_chunks / totalChunks) * 100,
);
setStatus({ kind: "uploading", progress });
if (chunkData.upload_status === "completed") {
setStatus({ kind: "done", fileUrl: chunkData.file_url });
return;
}
}
} catch (err) {
if ((err as Error).name === "AbortError") return; // user cancelled — silent
setStatus({ kind: "error", message: (err as Error).message });
}
}, []);
const cancel = useCallback(() => {
abortRef.current?.abort();
setStatus({ kind: "idle" });
}, []);
return { status, upload, cancel };
}The component
tsx
// components/ChunkUploader.tsx
import React, { useRef } from "react";
import { useChunkUpload } from "../hooks/useChunkUpload";
export function ChunkUploader() {
const { status, upload, cancel } = useChunkUpload();
const inputRef = useRef<HTMLInputElement>(null);
function handleChange(e: React.ChangeEvent<HTMLInputElement>) {
const file = e.target.files?.[0];
if (file) upload(file);
}
return (
<div style={{ fontFamily: "sans-serif", maxWidth: 480 }}>
<h2>Upload a file</h2>
<input
ref={inputRef}
type="file"
onChange={handleChange}
disabled={status.kind === "uploading"}
/>
{status.kind === "uploading" && (
<div style={{ marginTop: 12 }}>
<progress
value={status.progress}
max={100}
style={{ width: "100%" }}
/>
<p>{status.progress}% uploaded</p>
<button onClick={cancel}>Cancel</button>
</div>
)}
{status.kind === "done" && (
<p>
Done.{" "}
<a href={status.fileUrl} target="_blank" rel="noreferrer">
Download file
</a>
</p>
)}
{status.kind === "error" && (
<p style={{ color: "red" }}>Error: {status.message}</p>
)}
</div>
);
}Wiring it into your app
tsx
// App.tsx
import React from "react";
import { ChunkUploader } from "./components/ChunkUploader";
export default function App() {
return (
<main>
<ChunkUploader />
</main>
);
}A few things worth calling out:
file.slice(start, end) is the browser's native File API. It gives you a Blob for that byte range without reading the rest of the file into memory. You never hold more than one chunk in RAM at a time — which is precisely the point.
UploadStatus as a discriminated union means the component can use a status.kind check and TypeScript will narrow the type automatically. No isLoading, isError, isDone booleans scattered around — just one state value.
AbortController lets the user cancel mid-upload cleanly. The in-flight fetch gets aborted, the AbortError is caught and suppressed (it's not a real error), and the status resets to idle. If the user picks another file immediately, the previous upload is cancelled before the new one starts.
Retry per chunk means a network blip during chunk 7 of 20 only requires resending chunk 7. The rest are already saved on the server. Because the server is idempotent on duplicate chunk indices, retrying is always safe.
Parallel uploads: the loop above sends chunks sequentially. If you want faster uploads, you can batch with Promise.all — typically 3 to 5 concurrent chunks is a good ceiling before diminishing returns kick in. Just make sure to update progress correctly when multiple promises resolve at different times.
Choosing a Storage Backend for Production
Local disk is fine for development, demos, and single-server deployments with low volume. For anything beyond that, you will want an object storage service.
Here are the serious options, briefly:
AWS S3. The default choice for most teams. Mature SDK, excellent documentation, fine-grained IAM permissions, lifecycle rules, replication across regions. The Go SDK is well-maintained. If your infrastructure is already on AWS, this is almost always the right answer.
Google Cloud Storage. AWS S3's counterpart on GCP. Comparable features. If you are running on Google Cloud, this integrates naturally with IAM, Cloud Run, and other GCP services.
Cloudinary. Designed specifically for media files: images and videos. It handles resizing, format conversion, and CDN delivery out of the box. If your application involves image processing or video streaming, Cloudinary saves a significant amount of custom code. You upload once and request different sizes or formats via URL parameters.
ImageKit. Similar to Cloudinary in positioning. Real-time image transformation, CDN delivery, good free tier. Worth evaluating if Cloudinary's pricing is a concern.
Backblaze B2. An S3-compatible API at lower cost. The SDK is compatible with AWS S3 libraries. If raw storage cost matters and you do not need AWS-specific integrations, this is worth a look.
The key point: because this project defines a StorageProvider interface, adding any of these is a matter of writing a new adapter in internal/infrastructure/ that implements two methods: Store and Delete. The domain, application, and handler layers stay untouched.
For Cloudinary specifically, the adapter would look roughly like this:
go
func (a *CloudinaryAdapter) Store(assembledPath string, fileID uuid.UUID, originalName string) (string, string, error) {
resp, err := a.client.Upload.Upload(ctx, assembledPath, uploader.UploadParams{
PublicID: "chunk-uploads/" + fileID.String(),
ResourceType: "auto",
})
if err != nil {
return "", "", err
}
return resp.PublicID, resp.SecureURL, nil
}Then in main.go, replace localstorage.New(...) with your Cloudinary adapter constructor. That is the entire change.
Common Mistakes and How to Avoid Them
Not validating chunk_index on the server side. A client sending a negative index, or an index equal to or greater than total_chunks, can corrupt your chunk directory or cause out-of-bounds issues in assembly. The service validates this explicitly before writing anything to disk.
Not handling duplicate chunks. Network conditions can cause clients to retry a chunk that was already received. Without the isNewChunk check, the uploaded_chunks counter increments twice for the same index, and the server mistakenly thinks the file is complete before all unique chunks have arrived. The fix is a one-line os.Stat check before writing.
Assembling in filesystem order instead of indexed order. If you read chunk files using os.ReadDir and rely on sort order without zero-padding, you get chunk_1, chunk_10, chunk_2 ordering, which is wrong. Zero-pad to a fixed width and read by index in a deterministic loop.
Leaving orphaned chunks on disk. If an upload session starts but the client never sends all chunks (a user closes the browser, for example), the chunk directory stays on disk forever. In production, add a background job that deletes directories for sessions older than some threshold with a pending status.
Setting the BodyLimit too low. The chunk size the client uses must fit within the server's configured body limit. In this project, BodyLimit is set to 50 MB. If clients split files into 64 MB chunks, those requests will be rejected. Coordinate the chunk size between your client and server configuration.
Summary
Chunked file upload is not complicated in concept: split, send, reassemble. The complexity lies in handling the edge cases correctly: idempotency, ordering, failure states, and cleanup.
This implementation keeps responsibilities separated: the domain defines what a file is and what can go wrong, the application layer coordinates the chunking logic, infrastructure adapts to specific tools (PostgreSQL, local disk), and the HTTP layer translates between the application and HTTP clients.
The StorageProvider interface is the most important design decision in the whole project. It is a small surface area that decouples the chunking logic from the storage destination entirely. Wherever you decide to store files — local disk, S3, GCS, Cloudinary, or anywhere else — the chunk upload logic does not change.
That is, ultimately, what clean architecture is for.
Acknowledgment
This article was written as part of a hands-on discussion with Nugraha Aditama — exploring how chunked file upload actually works in practice, from the protocol level down to the Go implementation. The back-and-forth made the explanations sharper. Thank you.
Komentar(0)