Daniel Errante's Blog

Create an Image Tagger App with React.js and AI in Just a Few Simple Steps!

Daniel Errante — Wed, 08 Mar 2023 19:26:34 GMT

In this tutorial, we'll create a simple application that enables users to upload an image using a React.js form and display image tags using AI. We'll be using Next.js with TailwindCSS since it's straightforward to set up for both the front-end HTML and simple API endpoints with minimal configuration. We'll also be using Amazon's Rekognition API for the AI image tagging.

By the end of this tutorial, you will have an application that looks like this:

Prerequisites

To start with, we need to have an AWS access key and secret available to the local Next.js application. For this tutorial, we assume you already have the credentials. If you need instructions on obtaining those, you can go to this link - https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html. Once you have the credentials, ensure to export AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your shell environment locally. If you need help doing that, go to this link - https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html.

Building an image upload form

In my previous articles, I showed you how to build a file uploader that can support files up to 5 terabytes directly to Amazon S3 (https://blog.danoph.com/how-to-upload-large-files-directly-to-amazon-s3-in-reactnextjs). For this example, supporting 5TB file uploads seems a little overkill so we're going to start from scratch making the simplest thing possible that can accept images up to 5 megabytes. We're not going to store file uploads in a database or deploy the app anywhere. I just want to demonstrate how easy it is to build an image tagger using Amazon Rekognition AI.

Let's start by creating a new Next.js application with TailwindCSS preconfigured:

npx create-next-app -e with-tailwindcss image-classifier

After that's done, go into the new app directory and start the local server:

cd image-classifier
npm run dev

When you visit http://localhost:3000 your page should look like this:

Now, let's replace the default boilerplate component in pages/index.tsx with our own component that includes a file upload form:

import { useState } from 'react'
import type { NextPage } from 'next'
import Head from 'next/head'
import Image from 'next/image'

const Home: NextPage = () => {
  const onFileChanged = event => {
    const file = event.target.files[0];

    const uploadFile = async () => {
      const formData = new FormData();

      formData.append("file", file);

      const response = await fetch("/api/classify-image", {
        method: "POST",
        body: formData
      });

      console.log('response', response);
    }

    uploadFile();
  };

  return (
    "flex min-h-screen flex-col items-center justify-center py-2">
      
        Image Classifier
        "icon" href="/favicon.ico" />
      

      "flex w-full flex-1 flex-col items-center justify-center px-20 text-center">
        "text-6xl font-bold">
          Image Labeler
        

        "mt-6 flex max-w-4xl flex-wrap items-center justify-around sm:w-full">
          "file-upload"
            name="files"
            type="file"
            onChange={onFileChanged}
            accept="image/*"
          />
        
      
    
  )
}

export default Home

Your page should look like this when you go back to your browser:

The onChange on the file input calls the onFileChanged function in our component when you select a file and then it uploads the file to the /api/classify-image endpoint which we haven't created yet. Let's do that now.

Building the image classifier endpoint

We're going to be using the Amazon Rekognition API to label our images and the Formidable library to parse the form submission data from the upload form so we'll need to add two libraries to our project:

npm i @aws-sdk/client-rekognition formidable

Adding a new API endpoint is extremely easy in Next.js. All we have to do is name the file what we want the endpoint to be. Since our HTML form sends a POST request to /api/classify-image let's create a new file pages/api/classify-image.ts:

import type { NextApiRequest, NextApiResponse } from 'next'
import { RekognitionClient, DetectLabelsCommand } from "@aws-sdk/client-rekognition";
import { IncomingForm } from 'formidable';
import { readFileSync } from 'fs';

// we need to disable the default body parser since this endpoint is not accepting JSON
export const config = {
  api: {
    bodyParser: false,
  }
};

const client = new RekognitionClient({});

const getImageLabels = async (base64EncodedImage: Uint8Array) => {
  const response = await client.send(
    new DetectLabelsCommand({
      Image: {
        Bytes: base64EncodedImage
      }
    })
  );

  return response.Labels;
};

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  const imageBuffer = await (
    new Promise((resolve, reject) => {
      new IncomingForm().parse(req, (err, fields, files) => {
        resolve(readFileSync(files.file.filepath));
      });
    })
  );

  const labels = await getImageLabels(imageBuffer);

  res.status(200).json(labels);
}

A few things are going on here:

Near the top of the file, we are disabling the Next.js bodyParser so it doesn't try and parse the request body as JSON.
The getImageLabels function makes an API request to Amazon Rekognition with our base64 encoded image.
The default exported handler function is the actual Next.js API endpoint that our HTML upload form is calling. We didn't add any form validation or error handling so the function assumes an image file has been uploaded and encoded as multipart/form-data. IncomingForm's parse function doesn't return a promise (instead it expects a callback function) so we are wrapping that code with a new promise and resolving after the form is done parsing in the callback function. Formidable's IncomingForm parses the multipart form data and then saves the file to a temporary file path on disk by default so that's why we are reading the file from disk. Since readFileSync returns a Node.js Buffer we can pass that straight to the getImageLabels function and get our JSON label array back. After all that we respond to the front-end HTML component with the label data.

Changing the front end to display our image tags

Right now we aren't doing anything with the API response after we upload the file to our backend API endpoint besides logging to the browser console. Let's fix that by adding some state to keep track of the image labels that come back from the server.

First, import useState at the top of the file:

import { useState } from 'react'

And then let's add the state variable inside our component:

const [imageLabels, setImageLabels] = useState([]);

Now, we can use setImageLabels to capture the API response from the server inside our onFileChanged function. We can also remove the console.log that was there before so our function looks like this now:

  const onFileChanged = event => {
    const file = event.target.files[0];

    const uploadFile = async () => {
      const formData = new FormData();

      formData.append("file", file);

      const response = await fetch("/api/classify-image", {
        method: "POST",
        body: formData
      });

      setImageLabels(await response.json());
    }

    uploadFile();
  };

In the HTML view, let's display each label along with the confidence score that Rekognition gives us:

        {imageLabels.length > 0 && (
          "list" className="mt-6 max-w-md sm:w-full">
            {imageLabels.map((label, index) => (
              "py-2 flex items-center justify-between">
                
                  {label.Name}
                
                "text-lg font-bold">
                  {Math.round(label.Confidence)}%
                
              
            ))}
          
        )}

We are rounding the Confidence score is just for a nicer display since they are 8 or 9 decimal points.

It'd also be nice if we could see the image along with the labels that Rekognition returned. To keep things simple, let's add a preview image in the client-side HTML so we don't have to add any server-side code:

        {previewUrl && (
          "mt-6 h-96 aspect-w-10 aspect-h-7 block w-full overflow-hidden relative">
            "file uploader preview"
              src={previewUrl}
              fill
              sizes="(max-width: 768px) 100vw,
              (max-width: 1200px) 50vw,
              33vw"
              quality={100}
              className="pointer-events-none object-contain"
            />
          
        )}

Since we haven't declared previewUrl yet, let's add that to our state at the top of the component:

const [previewUrl, setPreviewUrl] = useState<string | null>(null);

And let's set the previewUrl after an image file has been selected in our onFileChanged function:

  const onFileChanged = event => {
    const file = event.target.files[0];

    setPreviewUrl(URL.createObjectURL(file));

    const uploadFile = async () => {
      const formData = new FormData();

      formData.append("file", file);

      const response = await fetch("/api/classify-image", {
        method: "POST",
        body: formData
      });

      setImageLabels(await response.json());
    }

    uploadFile();
  };

Side Note - If you've ever wanted to display images client side before they get uploaded to a server, this technique isn't dependent on React or Next.js.

So, after all of those changes, our pages/index.tsx file looks like this now:

import { useState } from 'react'
import type { NextPage } from 'next'
import Head from 'next/head'
import Image from 'next/image'

const Home: NextPage = () => {
  const [previewUrl, setPreviewUrl] = useState<string | null>(null);
  const [imageLabels, setImageLabels] = useState([]);

  const onFileChanged = event => {
    const file = event.target.files[0];

    setPreviewUrl(URL.createObjectURL(file));

    const uploadFile = async () => {
      const formData = new FormData();

      formData.append("file", file);

      const response = await fetch("/api/classify-image", {
        method: "POST",
        body: formData
      });

      setImageLabels(await response.json());
    }

    uploadFile();
  };

  return (
    "flex min-h-screen flex-col items-center py-2">
      
        Image Tagger
        "icon" href="/favicon.ico" />
      

      "flex w-full flex-1 flex-col items-center px-20">
        "text-6xl font-bold text-center">
          Image Tagger
        

        "mt-6 flex max-w-4xl flex-wrap items-center justify-around sm:w-full">
          "file-upload"
            name="files"
            type="file"
            onChange={onFileChanged}
            accept="image/*"
          />
        

        {previewUrl && (
          "mt-6 h-96 aspect-w-10 aspect-h-7 block w-full overflow-hidden relative">
            "file uploader preview"
              src={previewUrl}
              fill
              sizes="(max-width: 768px) 100vw,
              (max-width: 1200px) 50vw,
              33vw"
              quality={100}
              className="pointer-events-none object-contain"
            />
          
        )}

        {imageLabels.length > 0 && (
          "list" className="mt-6 max-w-md sm:w-full">
            {imageLabels.map((label, index) => (
              "py-2 flex items-center justify-between">
                
                  {label.Name}
                
                "text-lg font-bold">
                  {Math.round(label.Confidence)}%
                
              
            ))}
          
        )}
      
    
  )
}

export default Home

In the browser, click on Choose File and select an image file under 5 megabytes (that is the limit for Amazon Rekognition - https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-rekognition/interfaces/image.html). You might see a slight delay depending on the size of the image you uploaded, but after the file gets sent to Amazon Rekgonition and gets processed, you will see tags showing up for your uploaded image!

The full source code for this example is available here: https://github.com/danoph/ai-image-tagger

How To: Next.js Drag-and-Drop Image Uploads Directly to S3 and Displaying with CloudFront

Daniel Errante — Mon, 06 Mar 2023 18:01:08 GMT

In my first two tutorials on Next.js file uploads, we learned how to build a multiple file upload form in Next.js with drag-and-drop support and progress bars. Our files get stored in Amazon S3 but we aren't displaying the files anywhere on our page after the uploads are complete. In this tutorial, I'm going to change our multiple file uploader to only allow images to be uploaded and then we're going to display those images on our page using CloudFront and Amazon S3.

If you'd like to see the two previous articles, here they are:

And if you'd just like to see the source code for this tutorial, go here: https://github.com/danoph/file-uploader-demo/tree/story/cloudfront

By the end of this article, your application will look like this and display images from your S3 bucket we set up previously:

We're going to start from the point we left off in the previous article, https://blog.danoph.com/turning-our-react-single-file-s3-direct-uploader-into-a-multi-file-uploader-adding-progress-bars-and-drag-and-drop-support.

Changing our file uploader to only allow images

Ok, let's start by modifying our form's file input to only allow image uploads:

// pages/index.tsx

"file-upload"
  name="files"
  type="file"
  className="sr-only"
  onChange={onFilesChanged}
  multiple
  accept="image/*"
  value={inputValue}
/>

We should also check the mime type of the filename for our API endpoint in pages/api/multipart_uploads/index.ts and we'll introduce a third-party library called mime-types to help us out:

npm install mime-types

In pages/api/multipart_uploads/index.ts let's change our function to return an error if the upload is not an image:

// pages/api/multipart_uploads/index.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import { createMultipartUpload } from '@/lib/s3';
import mime from 'mime-types';

export default async function handler(req, res) {
  const { filename } = req.body;

  if (mime.lookup(filename).match(/image\//)) {
    const { uploadId, fileKey } = await createMultipartUpload({ filename });

    res.status(201).json({
      uploadId,
      fileKey,
    });
  } else {
    res.status(422).json({
      message: "Upload must be an image"
    });
  }
}

Setting up our CloudFront CDN

To display our images, we need to serve them over the web through a publicly available URL. I'm not going to assume you have a custom domain name to work with, so we can just use the default URL when we create our CloudFront distribution. If you'd like to see me walk you through how to use a custom domain name with a CloudFront distribution leave a comment and I'll write an article on how to do that.

Since we already have Terraform set up which I walked you through in the first article (https://blog.danoph.com/how-to-upload-large-files-directly-to-amazon-s3-in-reactnextjs) new infrastructure changes are going to be a lot easier than using the AWS Console. If you haven't already set up Terraform and run through the first article, go do that now.

We already have our Amazon S3 bucket set up in Terraform, so now we need to add a CloudFront distribution to serve files from that S3 bucket:

# main.tf

resource "aws_s3_bucket_public_access_block" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_cloudfront_distribution" "uploads" {
  origin {
    domain_name              = aws_s3_bucket.uploads.bucket_regional_domain_name
    origin_id                = "file-uploader-demo"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.uploads.cloudfront_access_identity_path
    }
  }

  enabled             = true

  default_cache_behavior {
    allowed_methods  = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "file-uploader-demo"

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "allow-all"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
  }

  viewer_certificate {
    cloudfront_default_certificate = true
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }
}

resource "aws_cloudfront_origin_access_identity" "uploads" {}

data "aws_iam_policy_document" "cloudfront_s3" {
  statement {
    actions   = ["s3:GetObject"]
    resources = ["${aws_s3_bucket.uploads.arn}/*"]

    principals {
      type        = "AWS"
      identifiers = [aws_cloudfront_origin_access_identity.uploads.iam_arn]
    }
  }
}

resource "aws_s3_bucket_policy" "uploads" {
  bucket = aws_s3_bucket.uploads.id
  policy = data.aws_iam_policy_document.cloudfront_s3.json
}

output "cloudfront_url" {
  value = aws_cloudfront_distribution.uploads.domain_name
}

Run terraform apply and note the cloudfront_url output that Terraform prints. We will need this when displaying our images and we also need to put this value in our next.config.js file to allow images coming from that domain:

/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  images: {
    remotePatterns: [
      {
        protocol: 'https',
        hostname: 'd1m5oohnuppcs9.cloudfront.net', // replace this with your cloudfront_url
      },
    ],
  },
}

module.exports = nextConfig

Since we changed the next.config.js file we will need to restart the server before being able to use Next.js Image in our code.

Creating an Image Gallery

Since we're using Tailwind CSS we can create a decent-looking image gallery without a ton of code and we get image optimization for free out of the box with Next.js. Let's add a new ImageGallery component to pages/index.tsx . We'll also need to add an import to the top of the file for next/image:

// pages/index.tsx

import Image from 'next/image';

const ImageGallery = ({ images }) => {
  return (
    "list" className="py-4 grid grid-cols-2 gap-x-4 gap-y-8 sm:grid-cols-3 sm:gap-x-6 lg:grid-cols-4 xl:gap-x-8">
      {images.map((image) => (
        "relative">
          "group h-48 aspect-w-10 aspect-h-7 block w-full overflow-hidden rounded-lg bg-gray-100 focus-within:ring-2 focus-within:ring-indigo-500 focus-within:ring-offset-2 focus-within:ring-offset-gray-100 relative">
            "(max-width: 768px) 100vw,
              (max-width: 1200px) 50vw,
              33vw"
              alt=""
              quality={100}
              className="pointer-events-none object-cover group-hover:opacity-75"
            />
            
          
          "pointer-events-none mt-2 block truncate text-sm font-medium text-gray-900">{image.title}
          "pointer-events-none block text-sm font-medium text-gray-500">
            { prettyBytes(image.size) }
          
        
      ))}
    
  )
}

Next let's use our new ImageGallery component inside of our HomeComponent and add some state to keep track of images that we'll need to pass into the ImageGallery:

// pages/index.tsx

export default function Home() {
  const [images, setImages] = useState([]);

And add the ImageGallery to the inside of HomeComponent's view:

// pages/index.tsx

  return (
    "mx-auto max-w-7xl sm:p-6 lg:p-8">

So your full HomeComponent should look like this:

// pages/index.tsx

export default function Home() {
  const [inputValue, setInputValue] = useState("");
  const [uploads, setUploads] = useState([]);
  const [draggingOver, setDraggingOver] = useState(false);

  const [images, setImages] = useState([]);

  const updateProgress = (filename, percentage) => {
    setUploads(state =>
      state.map(fileUpload =>
        fileUpload.uploader.file.name === filename
        ? { ...fileUpload, progress: percentage }
        : fileUpload
      )
    );
  };

  const addFiles = files => {
    setUploads(
      files.map(file => ({
        uploader: new Uploader({ file })
        .onProgress(({ percentage }) => {
          updateProgress(file.name, percentage);
        })
        .onComplete((uploadResponse) => {
          console.log('upload complete', uploadResponse);
        })
        .onError((error) => {
          console.error('upload error', error)
        }),
        progress: 0
      }))
    );
  };

  const onFilesChanged = e => {
    const files = [ ...e.target.files ];
    addFiles(files);
  };

  const uploadClicked = () => {
    if (!uploads.length) { return }

    uploads.forEach(upload => upload.uploader.start());
  };

  const stopEvent = e => {
    e.preventDefault();
    e.stopPropagation();
  }

  const handleDragEnter = e => {
    stopEvent(e);
  };

  const handleDragLeave = e => {
    stopEvent(e);
    setDraggingOver(false);
  };

  const handleDragOver = e => {
    stopEvent(e);
    setDraggingOver(true);
  };

  const handleDrop = e => {
    stopEvent(e);
    setDraggingOver(false);
    const files = [ ...e.dataTransfer.files ];
    addFiles(files);
  };

  return (
    "mx-auto max-w-7xl sm:p-6 lg:p-8">
      

      "flex text-sm text-gray-600">
        "w-full">
          `${draggingOver
              ? "border-blue-500"
              : "border-gray-300"
            } mt-1 flex items-center justify-center px-6 pt-5 pb-6 border-2 border-dashed rounded-md`}
            onDrop={handleDrop}
            onDragOver={handleDragOver}
            onDragEnter={handleDragEnter}
            onDragLeave={handleDragLeave}
          >
            "file-upload"
              className="inline-flex items-center rounded border border-transparent bg-indigo-100 px-2.5 py-1.5 text-xs font-medium text-indigo-700 hover:bg-indigo-200 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2"
            >
              "text-md">
                Choose File
              
              "file-upload"
                name="files"
                type="file"
                className="sr-only"
                onChange={onFilesChanged}
                multiple
                accept="image/*"
                value={inputValue}
              />
            
            "pl-1 text-sm">or drag and drop
          
        
      

      "py-2 text-sm text-gray-500">
        Any file up to 5TB
      

      {uploads.map(({ uploader: { file }, progress }) => (
        "py-2 flex flex-grow flex-col">
          "text-sm font-medium text-gray-900">
            { file.name }
          
          "text-sm text-gray-500">
            { file.type }
          
          "text-sm text-gray-500">
            { prettyBytes(file.size) }
          

          
        
      ))}

      
    
  )
}

If you haven't read my previous articles and are wondering what all the code is related to uploads, check out those articles first that I linked to at the top of the article.

Adding an API endpoint to fetch uploaded images

You might be wondering where our images are going to come from. Since we don't have a database yet, we can build a simple API endpoint to query files in our S3 bucket and return the images to the front end along with some metadata about the image like its filename, size, and CloudFront URL. Let's create pages/api/images.ts and add a handler function:

// pages/api/images.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import { listBucketFiles } from '@/lib/s3';
import mime from 'mime-types';

export interface UploadedImage {
  title: string;
  source: string;
  size: number;
}

// the S3 Object field types are defined as string | null so we have to be a little defensive when using
// those objects in our TypeScript code

export default async function handler(req, res: NextApiResponse) {
  const images = (await listBucketFiles())
  .filter(s3File => mime.lookup(s3File?.Key || "").match(/image\//))
  .map(s3File => ({
    title: s3File.Key || "some image",
    source: `https://d1m5oohnuppcs9.cloudfront.net/${s3File.Key}`,
    size: s3File.Size || 0,
  }));

  res.status(200).json(images);
}

You'll need to replace d1m5oohnuppcs9.cloudfront.net with your cloudfront_url from the Terraform output from earlier.

Some things to note:

I created a new TypeScript interface UploadedImage for what we expect in the front-end code to populate the ImageGallery component.
The Amazon javascript sdk v3 is a little cumbersome to work with because all the values for s3File can be undefined. That's why I added defaults to the title and size properties so TypeScript doesn't complain about s3File.Key and s3File.Size possibly being undefined.
I added a filter to the array of objects coming back from S3 to skip files that aren't images.

You'll notice that we are referencing a function called listBucketFiles that we haven't defined yet, so let's fix that now in the lib/s3.ts file:

// lib/s3.ts

export const listBucketFiles = async () => {
  const response = await client.send(
    new ListObjectsCommand({
      Bucket: UPLOAD_BUCKET,
    })
  );

  return response?.Contents || [];
}

We'll also need to add ListObjectsCommand to the list of imports at the top of lib/s3.ts:

// lib/s3.ts

import {
    S3Client,
    CreateMultipartUploadCommand,
    UploadPartCommand,
    CompleteMultipartUploadCommand,
    ListObjectsCommand,
} from "@aws-sdk/client-s3";

Fetching the images from our new API Endpoint

Now that we have an API endpoint set up to fetch images from s3 and transform each s3 object into an UploadedImage we can modify our HomeComponent to fetch images from the API. Let's add axios to the top of pages/index.tsx first:

// pages/index.tsx

import axios from "axios"

const API_BASE_URL = "/api/";

const api = axios.create({
  baseURL: API_BASE_URL,
});

And then add a useEffect in our HomeComponent to fetch the images from our API endpoint:

// pages/index.tsx

  useEffect(() => {
    const fetchImages = async () => {
      const response = await api.request({
        url: `/images`,
        method: "GET",
      })

      setImages(response.data);
    };

    fetchImages();
  }, []);

When you refresh the page you should now see images being displayed in the image gallery we just built:

Displaying images as they finish uploading

To display the images immediately after they're uploaded without refreshing the page, we'll need to modify our HomeComponent and the /api/multipart_uploads/[uploadId]/completions endpoint.

Let's start with the front end HomeComponent. After a file is done uploading, we want to take the response from the endpoint when an upload is finished and add that to our images array. Inside of addFiles where we are setting up our file uploads, change the onComplete function to this:

        .onComplete((newImage) => {
          setImages(state => [ newImage, ...state ]);
        })

TypeScript is going to complain that we didn't set a type for our images state so let's add a type to the array:

const [images, setImages] = useState([]);

We defined the UploadedImage interface in our pages/api/images.ts file but we are going to need it in another endpoint so let's create a new file lib/models.ts that can export this interface:

// lib/models.ts

export interface UploadedImage {
  title: string;
  source: string;
  size: number;
}

Then let's import UploadedImage so we can use it in our HomeComponent:

// pages/index.tsx

import { UploadedImage } from '@/lib/models';

Since we don't want to have this interface defined twice, remove the interface from pages/api/images.ts and add the same import statement to the top:

// pages/api/images.ts

import { UploadedImage } from '@/lib/models';

Now let's modify our endpoint that gets called when an upload is finished to complete the multipart upload so it returns an UploadedImage to the Next.js component:

// pages/api/multipart_uploads/[uploadId]/completions.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import { getS3ObjectMetadata, finishMultipartUpload } from '@/lib/s3';
import { UploadedImage } from '@/lib/models';

export default async function handler(req, res: NextApiResponse) {
  const { uploadId } = req.query;
  const { fileKey, parts } = req.body;

  const finishResponse = await finishMultipartUpload({
    fileKey,
    uploadId,
    parts
  });

  const objectMetadata = await getS3ObjectMetadata({ filename: fileKey });

  res.status(200).json({
    title: finishResponse.Key || "",
    source: `https://d1m5oohnuppcs9.cloudfront.net/${finishResponse.Key}`,
    size: objectMetadata.ContentLength || 0,
  });
}

Some things to note:

There is one minor problem with the CompleteMultipartUploadCommand because it does not return the size of the file as part of the response. That is why I am adding a new getS3ObjectMetadata function to lib/s3.ts to get the file size that I will show you in a minute.
The endpoint returns the same UploadedImage as the pages/api/images.ts endpoint.
You'll need to replace d1m5oohnuppcs9.cloudfront.net above with your cloudfront_url that we got a few steps earlier.

Now, let's add getS3ObjectMetadata to lib/s3.ts:

export const getS3ObjectMetadata = async ({ filename }) => {
  const response = await client.send(
    new HeadObjectCommand({
      Bucket: UPLOAD_BUCKET,
      Key: filename,
    })
  );

  return response;
}

We'll also need to import HeadObjectCommand so let's add that to the functions we are importing from @aws-sdk/client-s3:

import {
    S3Client,
    CreateMultipartUploadCommand,
    UploadPartCommand,
    CompleteMultipartUploadCommand,
    ListObjectsCommand,
    HeadObjectCommand,
} from "@aws-sdk/client-s3";

Now when you upload images they will each get added as they are finished as the first item in the image gallery:

Let me know if have any questions in the comments!

If you want to see the full source code for this tutorial go here: https://github.com/danoph/file-uploader-demo/tree/story/cloudfront

Master React File Uploading: Convert Single File Uploads into Multi File Uploads with Progress Bars and Drag-and-Drop Support in Just a Few Steps

Daniel Errante — Tue, 28 Feb 2023 19:30:12 GMT

This is a continuation of the previous article: How to Upload Large Files Directly to Amazon S3 in React/Next.js

Our "finished" React s3 direct uploader that we built has some limitations. We can only upload 1 file at a time, we don't have any progress indicators, and we would like to support dragging and dropping files into a drop area with some visual indicators. In this article, we will give the front end a little TLC with Tailwind CSS and expand our uploader to be able to handle many uploads at the same time.

Currently, our application looks like this:

By the end of this article, your application will look like this:

tldr; The full source code for this example is here: https://github.com/danoph/file-uploader-demo/tree/story/progress-and-multi-upload

I think the simplest thing to add first would be to add a little bit of styling as well as a progress bar for the single file uploader form. Let's add a progress bar component to our pages/index.tsx file:

// pages/index.tsx

const UploadProgressBar = ({ progress }) => (
  "py-5">
    "w-full bg-gray-200 rounded-full">
      `${
          progress === 0 ?
            'invisible'
            : ''
        } bg-indigo-600 text-xs font-medium text-blue-100 text-center p-0.5 leading-none rounded-full`}
        style={{ width: progress + "%" }}>
        {progress}%
      
    
  
);

Instead of immediately starting an upload when we choose a file, let's add an explicit button to start the upload and give a little less weight to our current upload button since it will be used to stage the file for upload. We can also add a few more details about the file selected like the file name, type and size. I want to show a human friendly size, so we can introduce a third party library called pretty-bytes that will convert bytes to KB, MB GB, etc:

npm i pretty-bytes

And reference it in our Home component. We will also need to add a useState to keep track of the file upload progress. We also need to modify the uploader.onProgress method so we can update progress as needed. Here is the modified Home component that uses our new UploadProgressBar component, keeps track of the current file upload progress, changes the button styling and adds a new button to be explicit about starting an upload:

// pages/index.tsx

export default function Home() {
  const [inputValue, setInputValue] = useState("");
  const [upload, setUpload] = useStatenull>(null);
  const [progress, setProgress] = useState(0);

  const onFileChanged = e => {
    const file = [ ...e.target.files ][0];

    const uploader = new Uploader({ file })
    .onProgress(({ percentage }) => {
      setProgress(percentage);
    })
    .onComplete((uploadResponse) => {
      console.log('upload complete', uploadResponse);
    })
    .onError((error) => {
      console.error('upload error', error)
    });

    setUpload(uploader);
  };

  const uploadClicked = () => {
    if (!upload) { return }

    upload.start();
  };

  return (
    "mx-auto max-w-7xl sm:p-6 lg:p-8">
      "flex text-sm text-gray-600">
        "file-upload"
          className="inline-flex items-center rounded border border-transparent bg-indigo-100 px-2.5 py-1.5 text-xs font-medium text-indigo-700 hover:bg-indigo-200 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2"
        >
          "text-md">
            Choose File
          
          "file-upload"
            name="files"
            type="file"
            className="sr-only"
            onChange={onFileChanged}
            value={inputValue}
          />
        
      

      "py-2 text-sm text-gray-500">
        Any file up to 5TB
      

      {upload && (
        "py-2 flex flex-grow flex-col">
          "text-sm font-medium text-gray-900">
            { upload.file.name }
          
          "text-sm text-gray-500">
            { upload.file.type }
          
          "text-sm text-gray-500">
            { prettyBytes(upload.file.size) }
          

          
        
      )}

      
    
  )
}

Before we can test this locally, we need to set up some ENV vars for our Next.js application like we did in Vercel. Let's create an .env.local file. This file is ignored in our .gitignore so we can put our secrets there. You will need to replace the upload_bucket variable with the bucket you created in the previous article.

bucket_region=us-east-1
upload_bucket=danoph-file-uploader-demo
access_key_id=$AWS_ACCESS_KEY_ID
access_key_secret=$AWS_SECRET_ACCESS_KEY

I noticed while writing this article that we could have run a command to pull the variables into our Vercel deployment, but then we would be using the same AWS access key ID and secret that we used to deploy our terraform. That would not be best practice security-wise so I'm kinda glad we didn't do it that way; however I think it is fine for local development.

Adding these values and restarting our Next.js application should allow us to upload a file locally directly to s3. After restarting the local server, choose a file and click the Upload File button. You should see upload progress now:

Side note - If you followed along with the previous article and you're seeing funky progress updates, I noticed there was a bug in the Uploader class from the previous article that had one small typo where part.partNumber should be part.PartNumber. Change line 228 of lib/Uploader.ts to look like this:

        const progressListener = this.handleProgress.bind(this, part.PartNumber - 1)

Adding drag and drop support

Adding drag and drop support is pretty simple. We don't need any external libraries but we will need to add some handlers in our component to handle the file drag, the file drop, showing when a file is being dragged over the drop area, and we will need to slightly refactor our onFileChanged to use a new function that we can also use for the drop file event. Here is what it will look like when we drag and drop a file onto the target:

Our Home component code looks like this now:

export default function Home() {
  const [inputValue, setInputValue] = useState("");
  const [upload, setUpload] = useStatenull>(null);
  const [progress, setProgress] = useState(0);
  const [draggingOver, setDraggingOver] = useState(false);

  const addFile = file => {
    const uploader = new Uploader({ file })
    .onProgress(({ percentage }) => {
      setProgress(percentage);
    })
    .onComplete((uploadResponse) => {
      console.log('upload complete', uploadResponse);
    })
    .onError((error) => {
      console.error('upload error', error)
    });

    setUpload(uploader);
  };

  const onFileChanged = e => {
    const file = [ ...e.target.files ][0];
    addFile(file);
  };

  const uploadClicked = () => {
    if (!upload) { return }

    upload.start();
  };

  const stopEvent = e => {
    e.preventDefault();
    e.stopPropagation();
  }

  const handleDragEnter = e => {
    stopEvent(e);
  };

  const handleDragLeave = e => {
    stopEvent(e);
    setDraggingOver(false);
  };

  const handleDragOver = e => {
    stopEvent(e);
    setDraggingOver(true);
  };

  const handleDrop = e => {
    stopEvent(e);
    setDraggingOver(false);
    const file = [ ...e.dataTransfer.files ][0];
    addFile(file);
  };

  return (
    "mx-auto max-w-7xl sm:p-6 lg:p-8">
      "flex text-sm text-gray-600">
        "w-full">
          `${draggingOver
              ? "border-blue-500"
              : "border-gray-300"
            } mt-1 flex items-center justify-center px-6 pt-5 pb-6 border-2 border-dashed rounded-md`}
            onDrop={handleDrop}
            onDragOver={handleDragOver}
            onDragEnter={handleDragEnter}
            onDragLeave={handleDragLeave}
          >
            "file-upload"
              className="inline-flex items-center rounded border border-transparent bg-indigo-100 px-2.5 py-1.5 text-xs font-medium text-indigo-700 hover:bg-indigo-200 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2"
            >
              "text-md">
                Choose File
              
              "file-upload"
                name="files"
                type="file"
                className="sr-only"
                onChange={onFileChanged}
                value={inputValue}
              />
            
            "pl-1 text-sm">or drag and drop
          
        
      

      "py-2 text-sm text-gray-500">
        Any file up to 5TB
      

      {upload && (
        "py-2 flex flex-grow flex-col">
          "text-sm font-medium text-gray-900">
            { upload.file.name }
          
          "text-sm text-gray-500">
            { upload.file.type }
          
          "text-sm text-gray-500">
            { prettyBytes(upload.file.size) }
          

          
        
      )}

      
    
  )
}

Some things to note:

The draggingOver state keeps track of when the file is being dragged over the drop area.
The onFileChanged calls a new function addFile which just takes a file as an input and sets the current upload to that file.
The handleDrop function uses the newly added addFile function and passes in the event's file.
The handleDragEnter, handleDragLeave, handleDragOver handle the styling of the drop area.
I added a new wrapper div around the file upload form part that triggers the drag events.

And that's it. Super simple to add drag and drop support to our existing file upload form.

Converting our single file uploader to a multi file uploader

It would be nice to allow people to drag and drop or select multiple files when uploading to our application. So we're going to make a few changes to allow for that. Let's change our upload state to uploads, add multiple to the file input in our view, and change our functions and variables to reflect the multiple uploads. We have another issue, where our progress state is tied to one upload. There are many different ways to do this, like having a separate array/hash for each upload's progress, modifying the Uploader class to keep track of its own progress (sometimes you can't do this if you're using external libraries), or making sure each upload in our array has progress alongside it by encapsulating the uploads in the array in a wrapper object, like this:

interface FileUpload {
  uploader: Uploader;
  progress: number;
}

export default function Home() {
    const [uploads, setUploads] = useState([]);

Let's choose the FileUpload path and modify all of our functions to handle multiple file uploads. So our Home component looks like this now:

interface FileUpload {
  uploader: Uploader;
  progress: number;
}

export default function Home() {
  const [inputValue, setInputValue] = useState("");
  const [uploads, setUploads] = useState([]);
  const [draggingOver, setDraggingOver] = useState(false);

  const updateProgress = (filename, percentage) => {
    setUploads(state =>
      state.map(fileUpload =>
        fileUpload.uploader.file.name === filename
        ? { ...fileUpload, progress: percentage }
        : fileUpload
      )
    );
  };

  const addFiles = files => {
    setUploads(
      files.map(file => ({
        uploader: new Uploader({ file })
        .onProgress(({ percentage }) => {
          updateProgress(file.name, percentage);
        })
        .onComplete((uploadResponse) => {
          console.log('upload complete', uploadResponse);
        })
        .onError((error) => {
          console.error('upload error', error)
        }),
        progress: 0
      }))
    );
  };

  const onFilesChanged = e => {
    const files = [ ...e.target.files ];
    addFiles(files);
  };

  const uploadClicked = () => {
    if (!uploads.length) { return }

    uploads.forEach(upload => upload.uploader.start());
  };

  const stopEvent = e => {
    e.preventDefault();
    e.stopPropagation();
  }

  const handleDragEnter = e => {
    stopEvent(e);
  };

  const handleDragLeave = e => {
    stopEvent(e);
    setDraggingOver(false);
  };

  const handleDragOver = e => {
    stopEvent(e);
    setDraggingOver(true);
  };

  const handleDrop = e => {
    stopEvent(e);
    setDraggingOver(false);
    const files = [ ...e.dataTransfer.files ];
    addFiles(files);
  };

  return (
    "mx-auto max-w-7xl sm:p-6 lg:p-8">
      "flex text-sm text-gray-600">
        "w-full">
          `${draggingOver
              ? "border-blue-500"
              : "border-gray-300"
            } mt-1 flex items-center justify-center px-6 pt-5 pb-6 border-2 border-dashed rounded-md`}
            onDrop={handleDrop}
            onDragOver={handleDragOver}
            onDragEnter={handleDragEnter}
            onDragLeave={handleDragLeave}
          >
            "file-upload"
              className="inline-flex items-center rounded border border-transparent bg-indigo-100 px-2.5 py-1.5 text-xs font-medium text-indigo-700 hover:bg-indigo-200 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2"
            >
              "text-md">
                Choose File
              
              "file-upload"
                name="files"
                type="file"
                className="sr-only"
                onChange={onFilesChanged}
                multiple
                value={inputValue}
              />
            
            "pl-1 text-sm">or drag and drop
          
        
      

      "py-2 text-sm text-gray-500">
        Any file up to 5TB
      

      {uploads.map(({ uploader: { file }, progress }) => (
        "py-2 flex flex-grow flex-col">
          "text-sm font-medium text-gray-900">
            { file.name }
          
          "text-sm text-gray-500">
            { file.type }
          
          "text-sm text-gray-500">
            { prettyBytes(file.size) }
          

          
        
      ))}

      
    
  )
}

We had to make a few changes but nothing very complicated. Some things to note:

The state variable uploads changed to an array of FileUpload objects which we defined in our new TypeScript interface.
We had to change the handleDrop and renamed onFileChanged to onFilesChanged to reflect having multiple uploads.
We added multiple to the file input to allow multiple files.
The addFiles function iterates through the array of dropped or selected files and sets the uploads to an array of FileUpload objects.
We added the updateProgress function that the Uploader onProgress callback uses to find an upload in the array by the file's name, then updating the progress for that specific one.
The uploadClicked changed to iterate through each upload and begin the upload.
The view changed slightly to iterate over each upload in the uploads array and display the same output like when we had a single upload.

You should be able to drag and drop or select multiple files and see progress for each individual file now:

And that's it for now!

The full source code for this example is here: https://github.com/danoph/file-uploader-demo/tree/story/progress-and-multi-upload

How to Upload Large Files Directly to Amazon S3 in React/Next.js

Daniel Errante — Fri, 17 Feb 2023 20:28:32 GMT

If you've ever needed to upload large files over 5 gigabytes to Amazon S3, you will run into a road block pretty quickly. Using multipart chunked file uploads is the way to get around that. Also, if you've built a file uploader before, you have to make the decision of whether the file upload should proxy through your API backend or go directly to S3. Proxying through your API backend is simpler to set up but has some drawbacks, such as tying up a server process during the file upload, and having to handle the file on the server's file system before sending it over to a file storage system like S3. It is also usually a lot slower than uploading directly to S3.

Uploading directly to s3 still requires some type of backend that can create presigned S3 URLs for the web front end since you don't want the front end being able to generate those on its own. Next.js provides a convenient way of doing that with API endpoints you can create that get deployed as edge functions.

In this tutorial, we are going to create a simple file uploader from scratch in Next.js using Tailwind CSS and TypeScript. We will also be using terraform to set up the necessary AWS resources to store the files we are uploading. We will also be deploying this app to Vercel. It sounds like a lot of moving parts but you can have everything working by following the steps in this tutorial in less than an hour.

The full source code for this application is here: https://github.com/danoph/file-uploader-demo

Create a new Next.js app with TypeScript:

npx create-next-app@latest file-uploader-demo --typescript --eslint
cd file-uploader-demo

Install Tailwind CSS:

Full instructions are here: https://tailwindcss.com/docs/guides/nextjs

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

I chose not to use the "src" directory or the experimental "app" directory in the prompts when setting up NextJS. So, we will need to change the tailwind.config.js file slightly:

/** @type {import('tailwindcss').Config} */
module.exports = {
  content: [
    "./pages/**/*.{js,ts,jsx,tsx}",
    "./components/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

In the styles/globals.css file, we want to remove the default NextJS styling and replace it with some tailwind modules, as well as a small styling hack that most of the TailwindUI templates need:

@tailwind base;
@tailwind components;
@tailwind utilities;

#__next {
  @apply flex flex-col h-full;
  // some tailwind templates need this line instead:
  // min-height: 100%;
}

Run the app locally:

npm run dev

Edit the main pages/index.tsx file and replace it with a simple Tailwind CSS component to make sure everything's working:

export default function Home() {
  return (
    "text-3xl font-bold underline">
      Hello world!
    
  )
}

When you visit the page in your browser the text should be large and underlined. I'm also going to add "noImplicitAny": false to the tsconfig.json file to get rid of annoying "No implicit any" TypeScript warnings and "downlevelIteration": true for a convenience function I'm using in a library we will see later in the article. So your tsconfig.json file should look like this:

{
  "compilerOptions": {
    "target": "es5",
    "lib": ["dom", "dom.iterable", "esnext"],
    "allowJs": true,
    "skipLibCheck": true,
    "strict": true,
    "forceConsistentCasingInFileNames": true,
    "noEmit": true,
    "esModuleInterop": true,
    "module": "esnext",
    "moduleResolution": "node",
    "downlevelIteration": true,
    "noImplicitAny": false,
    "resolveJsonModule": true,
    "isolatedModules": true,
    "jsx": "preserve",
    "incremental": true,
    "baseUrl": ".",
    "paths": {
      "@/*": ["./*"]
    }
  },
  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx"],
  "exclude": ["node_modules"]
}

Time to create the uploader

There are a few moving pieces to this part. We will need an upload form (the easy part) and 3 separate endpoints in our Next.js app to handle:

POST /api/multipart_uploads
Creating a "multipart upload" for S3. This gives you a unique file ID for the upload.
POST /api/multipart_uploads/{fileId}/part_url
Since we might be uploading large files, we want to split up the upload into separate chunks, so for each individual chunk, we need to generate a pre-signed URL for that chunk.
POST /api/multipart_uploads/{fileId}/completions
When all of the parts have been uploaded, we "complete" the multipart upload and AWS stitches together all of the chunks that we uploaded into a single file.

I am also going to use an Uploader TypeScript class that I found here: (https://github.com/pilovm/multithreaded-uploader/blob/master/frontend/uploader.js). I had to modify the Uploader to call our Next.js API endpoints that we will be making and also to overcome an obstacle with the original uploader that does not account for pre-signed URLs expiring if an upload takes longer than the expiration timeout for each multipart upload chunk URL. The class also uses axios so we need to add that library to our project:

npm i axios

Here is my uploader class that I modified to work with our Next.js app. Let's make a folder called lib and we will put this Uploader class in lib/Uploader.ts:

// lib/Uploader.ts

import axios from "axios"

const API_BASE_URL = "/api/";

const api = axios.create({
  baseURL: API_BASE_URL,
});

interface Part {
  ETag: string
  PartNumber: number
}

interface IOptions {
  chunkSize?: number;
  threadsQuantity?: number;
  file: File;
}

// original source: https://github.com/pilovm/multithreaded-uploader/blob/master/frontend/uploader.js
export class Uploader {
  chunkSize: number;
  threadsQuantity: number;
  file: File;
  aborted: boolean;
  uploadedSize: number;
  progressCache: any;
  activeConnections: any;
  parts: any[];
  uploadedParts: any[];
  uploadId: string | null;
  fileKey: string | null;
  onProgressFn: (progress) => void;
  onErrorFn: (err) => void;
  onCompleteFn: (response) => void;

  constructor(options: IOptions) {
    // this must be bigger than or equal to 5MB,
    // otherwise AWS will respond with:
    // "Your proposed upload is smaller than the minimum allowed size"
    this.chunkSize = options.chunkSize || 1024 * 1024 * 5
    // number of parallel uploads
    this.threadsQuantity = Math.min(options.threadsQuantity || 5, 15)
    this.file = options.file
    this.aborted = false
    this.uploadedSize = 0
    this.progressCache = {}
    this.activeConnections = {}
    this.parts = []
    this.uploadedParts = []
    this.uploadId = null
    this.fileKey = null
    this.onProgressFn = (progress) => console.log('progress', progress);
    this.onErrorFn = (err) => console.log('err', err);
    this.onCompleteFn = (response) => console.log('response', response);
  }

  start() {
    this.initialize()
  }

  async initialize() {
    try {
      const { data: { uploadId, fileKey } } = await api.request({
        url: "/multipart_uploads",
        method: "POST",
        data: {
          filename: this.file.name,
        },
      })

      this.uploadId = uploadId;
      this.fileKey = fileKey;

      const numberOfParts = Math.ceil(this.file.size / this.chunkSize)

      this.parts.push(
        ...[...Array(numberOfParts).keys()].map((val, index) => ({
          PartNumber: index + 1
        }))
      );

      this.sendNext();
    } catch (error) {
      await this.complete(error)
    }
  }

  sendNext() {
    const activeConnections = Object.keys(this.activeConnections).length

    if (activeConnections >= this.threadsQuantity) {
      return
    }

    if (!this.parts.length) {
      if (!activeConnections) {
        this.complete()
      }

      return;
    }

    const part = this.parts.pop();

    if (this.file && part) {
      const sentSize = (part.PartNumber - 1) * this.chunkSize
      const chunk = this.file.slice(sentSize, sentSize + this.chunkSize)

      const sendChunkStarted = () => {
        this.sendNext()
      }

      this.sendChunk(chunk, part, sendChunkStarted)
        .then(() => {
          this.sendNext()
        })
        .catch((error) => {
          this.parts.push(part)
          this.complete(error)
        })
    }
  }

  // terminating the multipart upload request on success or failure
  async complete(error: unknown | undefined = null) {
    if (error && !this.aborted) {
      this.onErrorFn(error)
      return
    }

    if (error) {
      this.onErrorFn(error)
      return
    }

    try {
      const response = await this.sendCompleteRequest()
      this.onCompleteFn(response);
    } catch (error) {
      this.onErrorFn(error)
    }
  }

  // finalizing the multipart upload request on success by calling
  // the finalization API
  async sendCompleteRequest() {
    if (this.uploadId && this.fileKey) {
      const response = await api.request({
        url: `/multipart_uploads/${this.uploadId}/completions`,
        method: "POST",
        data: {
          fileKey: this.fileKey,
          parts: this.uploadedParts,
        },
      })

      return response.data;
    }
  }

  sendChunk(chunk, part, sendChunkStarted): Promise<void> {
    return new Promise((resolve, reject) => {
      this.upload(chunk, part, sendChunkStarted)
        .then((status) => {
          if (status !== 200) {
            reject(new Error("Failed chunk upload"))
            return
          }

          resolve()
        })
        .catch((error) => {
          reject(error)
        })
    })
  }

  // calculating the current progress of the multipart upload request
  handleProgress(part, event) {
    //console.log('part', part, 'event', event);
    if (this.file) {
      if (event.type === "progress" || event.type === "error" || event.type === "abort") {
        this.progressCache[part] = event.loaded
      }

      if (event.type === "uploaded") {
        this.uploadedSize += this.progressCache[part] || 0
        delete this.progressCache[part]
      }

      const inProgress = Object.keys(this.progressCache)
        .map(Number)
        .reduce((memo, id) => (memo += this.progressCache[id]), 0)

      const sent = Math.min(this.uploadedSize + inProgress, this.file.size)

      const total = this.file.size

      const percentage = Math.round((sent / total) * 100)

      this.onProgressFn({
        sent: sent,
        total: total,
        percentage: percentage,
      })
    }
  }

  upload(file, part, sendChunkStarted) {
    return new Promise(async (resolve, reject) => {
      if (this.uploadId && this.fileKey) {
        // we need to get the multipart chunk url immediately before starting the upload
        // since creating them beforehand may result in the urls expiring
        const { data: { signedUrl } } = await api.request({
          url: `/multipart_uploads/${this.uploadId}/part_url`,
          method: "POST",
          data: {
            fileKey: this.fileKey,
            partNumber: part.PartNumber,
          }
        })

        // - 1 because PartNumber is an index starting from 1 and not 0
        const xhr = (this.activeConnections[part.PartNumber - 1] = new XMLHttpRequest())

        sendChunkStarted()

        const progressListener = this.handleProgress.bind(this, part.PartNumber - 1)

        xhr.upload.addEventListener("progress", progressListener)

        xhr.addEventListener("error", progressListener)
        xhr.addEventListener("abort", progressListener)
        xhr.addEventListener("loadend", progressListener)

        xhr.open("PUT", signedUrl)

        xhr.onreadystatechange = () => {
          if (xhr.readyState === 4 && xhr.status === 200) {
            // retrieving the ETag parameter from the HTTP headers
            const ETag = xhr.getResponseHeader("etag")

            if (ETag) {
              const uploadedPart = {
                PartNumber: part.PartNumber,
                // removing the " enclosing carachters from
                // the raw ETag
                ETag: ETag.replaceAll('"', ""),
              }

              this.uploadedParts.push(uploadedPart)

              resolve(xhr.status)
              delete this.activeConnections[part.PartNumber - 1]
            }
          }
        }

        xhr.onerror = (error) => {
          console.log('xhr error', error);
          reject(error)
          delete this.activeConnections[part.PartNumber - 1]
        }

        xhr.onabort = () => {
          console.log('xhr abort');
          reject(new Error("Upload canceled by user"))
          delete this.activeConnections[part.PartNumber - 1]
        }

        xhr.send(file)
      }
    })
  }

  onProgress(onProgress) {
    this.onProgressFn = onProgress
    return this
  }

  onComplete(onComplete) {
    this.onCompleteFn = onComplete
    return this
  }

  onError(onError) {
    this.onErrorFn = onError
    return this
  }

  abort() {
    Object.keys(this.activeConnections)
      .map(Number)
      .forEach((id) => {
        this.activeConnections[id].abort()
      })

    this.aborted = true
  }
}

After adding this file, let's create an upload form and wire up the front end to use this new Uploader class.

// pages/index.tsx

import { useState } from 'react';
import { Uploader } from '@/lib/Uploader';

export default function Home() {
  const [inputValue, setInputValue] = useState("");
  const [upload, setUpload] = useStatenull>(null);

  const onFileChanged = e => {
    const file = [ ...e.target.files ][0];
    const uploader = new Uploader({ file })
    .onProgress(({ percentage }) => {
      console.log('upload progress', percentage);
    })
    .onComplete((uploadResponse) => {
      console.log('upload complete', uploadResponse);
    })
    .onError((error) => {
      console.error('upload error', error)
    });

    setUpload(uploader);

    uploader.start();
  };

  return (
    "mx-auto max-w-7xl sm:p-6 lg:p-8">
      "flex text-sm text-gray-600">
        "file-upload"
          className="inline-flex items-center rounded-md border border-transparent bg-indigo-600 px-4 py-2 text-sm font-medium text-white shadow-sm hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:ring-offset-2"
        >
          "text-lg">
            Upload a file
          
          "file-upload"
            name="files"
            type="file"
            className="sr-only"
            onChange={onFileChanged}
            value={inputValue}
          />
        
      

      "py-2 text-sm text-gray-500">
        Any file up to 5TB
      
    
  )
}

Your page should look like this:

Now we need to add those 3 endpoints for creating a multipart upload, creating a multipart part upload url, and completing the multipart upload. We will need to add two npm packages to our project: @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner:

npm i @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

To simplify creating our 3 endpoints, I've created a few TypeScript functions that will assist us. We will put this file in lib/s3.ts:

// lib/s3.ts

import {
    S3Client,
    CreateMultipartUploadCommand,
    UploadPartCommand,
    CompleteMultipartUploadCommand,
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const REGION = process.env.bucket_region;
const UPLOAD_BUCKET = process.env.upload_bucket;

// NOTE: these are named differently than the normal AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
// because Vercel does not allow you to set those environment variables for a deployment
const client = new S3Client({
  region: REGION,
  credentials: {
    accessKeyId: `${process.env.access_key_id}`,
    secretAccessKey: `${process.env.access_key_secret}`,
  }
});

export const createMultipartUpload = async ({ filename }) => {
  const { Key, UploadId } = await client.send(
    new CreateMultipartUploadCommand({
      Bucket: UPLOAD_BUCKET,
      Key: filename,
      ACL: "private",
    })
  );

  return {
    uploadId: UploadId,
    fileKey: Key,
  }
}

export const createMultipartUploadPart = async ({ fileKey, uploadId, partNumber }) => {
  const command = new UploadPartCommand({
    Bucket: UPLOAD_BUCKET,
    Key: fileKey,
    UploadId: uploadId,
    PartNumber: partNumber,
  });

  const signedUrl = await getSignedUrl(
    client as any, // avoiding typescript lint errors
    command as any, // avoiding typescript lint errors
    {
      expiresIn: 3600,
    }
  );

  return {
    signedUrl
  }
}

export const finishMultipartUpload = async ({ fileKey, uploadId, parts }) => {
  const response = await client.send(
    new CompleteMultipartUploadCommand({
      Bucket: UPLOAD_BUCKET,
      Key: fileKey,
      UploadId: uploadId,
      MultipartUpload: {
        Parts: parts.sort((a, b) => {
          if (a.PartNumber < b.PartNumber) {
            return -1;
          }

          if (a.PartNumber > b.PartNumber) {
            return 1;
          }

          return 0;
        })
      }
    })
  );

  return response;
};

Now we can create our endpoints that will end up using these functions. Let's start with creating the multipart upload. We can try and keep this as RESTful as possible by creating the folder pages/api/multipart_uploads and then creating an index.ts file within that folder. This will handle creating the multipart upload:

// pages/api/multipart_uploads/index.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import { createMultipartUpload } from '@/lib/s3';

export default async function handler(req, res) {
  const { filename } = req.body;
  const { uploadId, fileKey } = await createMultipartUpload({ filename });

  res.status(201).json({
    uploadId,
    fileKey,
  });
}

Next, let's create the endpoint for creating a multipart upload chunk url. Let's make a folder called [uploadId] inside the multipart_uploads folder and name the file part_url.ts:

// pages/api/multipart_uploads/[uploadId]/part_url.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import { createMultipartUploadPart } from '@/lib/s3';

export default async function handler(req, res) {
  const { uploadId } = req.query;
  const { fileKey, partNumber } = req.body;

  const { signedUrl } = await createMultipartUploadPart({
    fileKey,
    uploadId,
    partNumber
  });

  res.status(201).json({
    signedUrl
  });
}

And finally let's create the endpoint for completing a multipart upload in pages/api/multipart_uploads/[uploadId]/completions.ts:

// pages/api/multipart_uploads/[uploadId]/completions.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import { finishMultipartUpload } from '@/lib/s3';

export default async function handler(req, res) {
  const { uploadId } = req.query;
  const { fileKey, parts } = req.body;

  const response = await finishMultipartUpload({
    fileKey,
    uploadId,
    parts
  });

  res.status(200).json({});
}

These endpoints will not work out of the box because we need to set up our S3 bucket. Let's start writing some Terraform!

In order to create terraform for our new app, we need to manually create an s3 bucket that will house our terraform config. So, go into S3 in the AWS Console (https://aws.amazon.com) and create a new bucket and make sure Object Versioning is enabled. Since S3 bucket names have to be unique across all of AWS, I usually put my username or the app domain in the front of the bucket name to ensure it's unique. Don't use dots in the name of the bucket because that has other consequences that we don't want to run into for this tutorial. So, for example, for this application I am going to name my bucket danoph-file-uploader-demo-terraform and pick us-east-1 for the region.

In order to run terraform locally, make sure you have Terraform installed. On MacOS you can run brew install terraform or use tfenv if you want to manage multiple versions. Make sure the version you are using is near 1.3.8, which is the latest version as of this writing. You can check your version by running terraform version

Also, you need to make sure you have programmatic access to AWS. If you are logged in as the root user in your AWS account, you can click on your name in the top right, and then click Security Credentials. Under the Access Keys section, click on Create access key. Make sure to download these credentials since you will only see them once. If you are more familiar with AWS, you probably know giving the root AWS user isn't the most secure thing to do. The more secure way to do this would be to create a new IAM user that we will use to run terraform locally, and restricting access to only the certain resources we need to manage in AWS. That is outside the scope of this article so I'm going to assume you have created your access key and secret and now you need to put them in your ~/.bash_profile or ~/.zshrc file.

export AWS_ACCESS_KEY_ID="{value here}"
export AWS_SECRET_ACCESS_KEY="{value here}"

Make sure after adding these values to resource your ~/.zshrc or ~/.bash_profile by running . ~/.zshrc or . ~/.bash_profile.

Now that you have credentials set up, let's create some boilerplate terraform inside a main.tf file:

# main.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "4.39.0"
    }
  }
}

terraform {
  backend "s3" {
    bucket = "danoph-file-uploader-demo-terraform"
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

You'll need to replace danoph-file-uploader-demo-terraform with the bucket you just created manually to house our terraform state.

Now, if you run terraform init you should see something similar to this output:

We also want to make sure to add the .terraform folder to our .gitignore:

echo '.terraform' >> .gitignore

Now we can get onto the fun stuff. We need to create a few things in AWS for our app to work:

An IAM user with an access key and secret that our app will use when it's deployed to Vercel.
An S3 bucket that we will use for our file uploads
Some permissions and policies around our app being able to work with our s3 bucket

# main.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "4.39.0"
    }
  }
}

terraform {
  backend "s3" {
    bucket = "danoph-file-uploader-demo-terraform"
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
}

locals {
  upload-bucket-name = "danoph-file-uploader-demo"
}

resource "aws_s3_bucket" "uploads" {
  bucket = local.upload-bucket-name
}

resource "aws_s3_bucket_server_side_encryption_configuration" "uploads" {
  bucket = aws_s3_bucket.uploads.bucket

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "AES256"
    }
  }
}

resource "aws_s3_bucket_cors_configuration" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  cors_rule {
    allowed_headers = ["*"]
    allowed_methods = ["GET", "PUT", "POST"]
    allowed_origins = ["*"]
    expose_headers  = ["ETag"]
  }
}

resource "aws_iam_user" "vercel" {
  name = "file-uploader-demo"
  path = "/system/"
}

resource "aws_iam_access_key" "vercel" {
  user = aws_iam_user.vercel.name
}

resource "aws_iam_policy" "s3_access" {
  name        = "vercel_file_uploader_demo"
  path        = "/"
  description = "IAM policy for s3 access from vercel"

  policy = <"Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::${local.upload-bucket-name}/*",
      "Effect": "Allow"
    }
  ]
}
EOF
}

resource "aws_iam_user_policy_attachment" "s3_access" {
  user       = aws_iam_user.vercel.name
  policy_arn = aws_iam_policy.s3_access.arn
}

output "access_key_id" {
  value = aws_iam_access_key.vercel.id
}

output "access_key_secret" {
  sensitive = true
  value = aws_iam_access_key.vercel.secret
}

To see what's going to be created in AWS, we can run the following command to do a dry run in terraform:

terraform plan

You will see the s3 bucket, IAM user and policy, etc. about to be created. When you're ready to make the changes, run:

terraform apply

It will ask you if you're sure you want to make these changes by typing yes. When I am using terraform in the real world, I make sure to review the plan before typing yes whenever I am making changes. Many times I have seen people not look closely enough at what terraform is about to do and they end up accidentally deleting a production database server, which can cause weeks worth of work if there aren't backups or redundancy set up. That's just a tip from me for your future terraform endeavors to review the plan before actually typing yes :)

Anyway, when the vercel IAM user gets created when running this terraform, make sure to copy the access key id somewhere safe. Since we marked the access_key_secret output as sensitive, we need to explicitly run this command to see the output value:

terraform output access_key_secret

Now that we have everything set up in AWS, let's switch over to Vercel and get everything set up over there. We will need to make sure our app is pushed up to GitHub first. After that, we can visit vercel and import our new repository:

Click on "Import" next to our repository. On the next page, we will need to add 4 environment variables that our lib/s3.ts file is referencing:

bucket_region
upload_bucket
access_key_id
access_key_secret

Vercel will automatically inject these values into our application that our application is referencing using process.env. This is good practice to avoid hardcoding secrets and passwords in applications. You'd be shocked to know how many projects I've seen that hardcode usernames and passwords directly into the codebase.

I purposely am not showing the values for access_key_id and access_key_secret since they are displayed in plain text. Just make sure to enter all 4 values and then click "Deploy"

After the successful deployment, you should be able to visit the domain vercel assigns your app:

Once you visit the app, open up the developer console so you can see upload progress when uploading a file. Then click on the "Upload a file" button and watch the developer console for the upload to finish.

Now, if you head over to your S3 bucket in the AWS Console, you should see a new file has just arrived.

And there you have multipart chunked uploading directly to Amazon S3 from scratch!

If people are interested in knowing more about this topic, let me know in the comments. Since we aren't displaying the images on the page currently, we could walk through a way to display the files served through a CDN like CloudFront, or we could possibly explore expanding our single file uploader into a multiple file uploader.

The full source code for this application is here: https://github.com/danoph/file-uploader-demo

Daniel Errante's Blog

Create an Image Tagger App with React.js and AI in Just a Few Simple Steps!

Prerequisites

Building an image upload form

"text-6xl font-bold"> Image Labeler

Building the image classifier endpoint

Changing the front end to display our image tags

"text-6xl font-bold text-center"> Image Tagger

How To: Next.js Drag-and-Drop Image Uploads Directly to S3 and Displaying with CloudFront

Changing our file uploader to only allow images

Setting up our CloudFront CDN

Creating an Image Gallery

Adding an API endpoint to fetch uploaded images

Fetching the images from our new API Endpoint

Displaying images as they finish uploading

Master React File Uploading: Convert Single File Uploads into Multi File Uploads with Progress Bars and Drag-and-Drop Support in Just a Few Steps

Adding drag and drop support

Converting our single file uploader to a multi file uploader

How to Upload Large Files Directly to Amazon S3 in React/Next.js

Create a new Next.js app with TypeScript:

Install Tailwind CSS:

"text-3xl font-bold underline"> Hello world!

Time to create the uploader