Multi-file Uploads Using Next.js

Multi-file Uploads Using Next.js 13 Serverless Functionality, Express 4, and Amazon S3 Pre-signed URLs in TypeScript and Node.js 18A tutorial on uploading multiple files using Next.js 13’s serverless functionality, Express 4, and Amazon S3 pre-signed URLs in TypeScript and Node.jsWe needed the ability to have a file upload client component for our application. The ask is straightforward: allow the user to upload file(s) from their computer to an S3 bucket.At first, the solution seemed pretty simple to understand: make a component that submits files to our NextJS API endpoint (hosted on Vercel) and then post that data to our server running Express in the cloud (hosted on Heroku). We could in theory go right from the NextJS backend, but we use the server as a proxy in a few other instances, so to maintain consistency we use the same path.One main problem with this initial approach is found when trying to submit files on a front-end hosted in a serverless style (Vercel). You will notice limits on file sizes and request sizes. In our case, going to NextJS Vercel had a limit of a few MBs. This meant we couldn’t stream a file after a certain size. This caused us to pivot to the Amazon S3 pre-signed URL approach, which comes directly from the browser.Some areas I would recommend reading up on that helped me solve this problem:How NodeJS handles HTTP interactions (specifically what/how streams work)Piping from one request to anotherUnderstanding what multipart/form-data meansComparing in-memory storage / storing locally to streaming files.Serverless functionality and its limitationsPre-signing URLs with S3.The requirements I set for myself were:No local storing of the file.Handle multiple file uploads.Stay as close to vanilla NodeJS as possible.I want things to be fast, efficient, and clean and have an understanding instead of just “magic” going on. I am going to break this into a few different sections, to help show my thought process and my journey to the solution — which works, by the way :).How to do itHere is a simple diagram I created to show my thought process. We have a button, onSubmit will call the NextJS back-end. That makes a call to our server, which calls S3. S3 returns a pre-signed URL. We use that URL to POST to.Asks for a pre-signed URL, returns to front-end, POST files directly to S3 from browser.Our component is pretty straightforward. It is just a button with the above architecture wired up:"use client";import { Button, Input, Typography } from "@mui/material";import { FormEvent, useState } from "react";import { useSnackbar } from "notistack";export default function UploadFileButton(props: { getFileList: Function; currentDir: string;}) { const { getFileList, currentDir } = props; const [files, setFiles] = useState<FileList | null>(); const { enqueueSnackbar } = useSnackbar(); function handleFiles( e: React.ChangeEvent<HTMLTextAreaElement | HTMLInputElement> ) { const files = (e.target as HTMLInputElement).files; setFiles(files); } async function handleSubmit(e: FormEvent<HTMLFormElement>) { e.preventDefault(); if (!files) { return; } if (files.length === 0) { return; } for (const index in files) { const file = files[index]; if (!(file instanceof File)) { continue; } const fileName = file.name; // Get signed key for file name in current directory. const response = await fetch( `/api/presignedurl?fileName=${currentDir.slice(1)}${fileName}` ); const data = await response.json(); const url = data.data.url; fetch(url, { method: "PUT", body: file, }) .then((response) => { if (!response.ok) { return enqueueSnackbar( `Error uploading ${fileName} to ${currentDir}.`, { variant: "error", } ); } enqueueSnackbar( `${fileName} successfully uploaded to ${currentDir}.`, { variant: "success", } ); getFileList(); }) .catch((err) => { enqueueSnackbar(`Error uploading ${fileName} to ${currentDir}.`, { variant: "error", }); }); } } return ( <> <form method="POST" onSubmit={handleSubmit} encType="multipart/form-data"> <Input onChange={handleFiles} type="file" inputProps={{ multiple: true, accept: ".gcode" }} /> <Button variant="outlined" sx={{ border: "1px solid black", color: "black", ml:1 }} size="small" type="submit" > Submit </Button> </form> <Typography variant="caption">*.gcode file extension only</Typography> </> );}Our server code is quite simple as well:// Use the presigner libraryimport { getSignedUrl } from "@aws-sdk/s3-request-presigner";// Somewhere in expressthis.app.get('/api/presignedurl', async (req: Request, res: Response) => { const fileName = String(req.query.fileName); const preSignedUrl = await getPresignedUrl(fileName) const signedUrlObject = { url: preSignedUrl } // Custom success response message. return res.send(new SuccessResponse(20000, signedUrlObject)) });// Presigned URL functionpublic async getPresignedUrl(fileName: string) { const bucketParams: PutObjectCommandInput = { Bucket: process.env.AWS_S3_BUCKET, Key: fileName, ContentType: "multipart/form-data", }; const command = new PutObjectCommand(bucketParams); const signedUrl = await getSignedUrl(this.s3Client, command, { expiresIn: 300, }); return signedUrl; }💡 If you find yourself needing to use the Pre-signed URL function over and over for other projects, you could consider extracting the code from your codebase into a package and share it across multiple projects with the help of an open-source tool like Bit.Learn more here:Extracting and Reusing Pre-existing Components using bit addA pre-signed URL is a temporary URL given to a user where they can write an object to S3. Instead of having an API key with write access, you specify exactly where that file is going to live, and then Amazon returns a URL where you can POST to. This URL is only available for a certain amount of time, and cannot be used afterwards.Once we have our pre-signed URL, we are able to make a POST request directly to the URL. One benefit of this approach is that the user is uploading directly from their browser. In some other approaches, you may see a Stream being used on the server, and that opens a stream to the bucket. This is a valid approach, but I wanted to go right from the client to the bucket, so the Server didn’t have to store anything. Finally, we enqueue a snack bar using Notistack (not necessary) to show the user that their file has been uploaded successfully.This was the most elegant solution I could come up with that accomplished the criteria that I listed above. If you have any inputs or insights, I would be interested in seeing different approaches.Build Apps with reusable components, just like LegoBit’s open-source tool help 250,000+ devs to build apps with components.Turn any UI, feature, or page into a reusable component — and share it across your applications. It’s easier to collaborate and build faster.→ Learn moreSplit apps into components to make app development easier, and enjoy the best experience for the workflows you want:→ Micro-Frontends→ Design System→ Code-Sharing and reuse→ MonorepoLearn more:Creating a Developer Website with Bit componentsHow We Build Micro FrontendsHow we Build a Component Design SystemHow to reuse React components across your projects5 Ways to Build a React MonorepoHow to Create a Composable React App with BitHow to Reuse and Share React Components in 2023: A Step-by-Step Guide5 Tools for Building React Component Libraries in 2023Multi-file Uploads Using Next.js was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.