Secure file management with AWS, VueJS and Laravel

Introduction

In my day job, we store thousands upon thousands of hours of regulatory video data. About 8 years ago we were automatically transferring these videos over a WiFi network that our systems connected to whenever they came within range. Video and GPS data was combined, compressed and sent back to our office over a VPN. It was cool, it was state of the art and it was cheaper than using cell data or physical media.

Our crazy competitors were still mailing physical hard drives around. In an effort to encourage competition, by the time the trial turned in to a proper programme, the government regulators deemed that physically transferring USB drives was the way to go. We’re still scratching our heads on that one.

We begrudgingly tore down our WiFi and retrofitted our secure, elegant solution with the Pony Express.

Skip forwards a few years and things are rapidly changing. We no longer store data in our office, it’s all in AWS S3 because we write software; we don’t manage hard drives. Video is no longer H.264 encoded; we’re up to H.266 now, which is truly amazing in terms of reduction in storage and increase in quality. Much of the video was like watching paint dry… nothing to see here, so AI is being introduced to both bookend the parts of the video needing to be reviewed and automatically annotate the video, reducing the costs to our customers.

But the issue of transferring that media is still a problem.

“Wouldn’t it be nice to have a drag and drop media management system that is secure too?”. That was the question that started it all… and like all such ideas, it grew and became a monster. That’s why this is a series of articles rather than one. I’ll break it down in to a number of parcels of work allowing us to get deeper into the thought processes that make this thing work, and yes… it does work!

This is a full stack solution. The pieces that make up this puzzle from front to back are VueJS, InertiaJS, Laravel along with Amazon S3, Cognito, IAM and CloudFront.

In this article, I will describe what it is we want to achieve and we will explain setting up the Amazon side of things

In part 2, we will delve into some front end code with VueJS and in part 3 we will talk about using Laravel and zip it all together.

In the beginning…

In applications, it is common to provide users with the ability to upload data. These are files such as documents, or media such as videos, photos or sound. Typically the process looks something like this:

1. the user uploads the file to the web application server
2. the web application server saves the file in a temporary space for processing and manipulation
3. the web application server stores the file in persistent memory

Our application servers typically run some kind of PHP framework. The PHP framework takes care of all of the nitty gritty of dealing with databases, filesystems etc. and allows us as developers to move onto the other things that make our web apps brilliant. We’ve been proponents of Laravel for the past 8 years and think that Taylor Otwell and the team at Laravel have done an amazing job.

While the process is simple, (Laravel provides a powerful filesystem abstraction), it’s not very efficient. Why would you upload media to your web server only to then transfer it again to your persistent storage? Media uploads are typically large, so these can really eat up your network I/O, CPU and local storage. How do you handle failures and retries on the ingress and egress and how is your website at handling thousands of users doing multiple uploads around the same time? Your only option is to scale everything out and ensure that there is sufficient network, memory and CPU resources available.

Retrieving User Uploads

In higher volume applications, it’s always good to split your media resources from your code as it allows you to scale massively. AWS S3 is highly available and durable, making it an ideal persistent store for user uploads, but it doesn’t support the significant performance advantages to getting our media via HTTP/2+. For MP4 videos that’s on par with getting your MOOV ATOM right at the start of the file…, and we are definitely doing this for performance.

The monster just grew more heads.

The browser now consumes media for our application from AWS CloudFront (think big pipes between S3 and the CloudFront servers, mixed with regional caching, delivery to the client over HTTPS/2 and we think you’re already onto a winner - We are still waiting on HTTPS/3 support with CloudFront).

The URL of the media in the CloudFront Cache is stored in our application database and retrieved either as part of a page load, page refresh or on demand via an AXIOS type call. The browser then handles fetching that media and those media requests never come anywhere near our application server freeing up our resources for other requests and processing. Furthermore, we can take advantage of the significant investment AWS has made in securing CloudFront resources and protecting them from DDOS type attacks whilst at the same time reducing the surface area of our own application server to attack from the Internet.

Storing, Changing and Deleting User Uploads

All other media operations with the user media will be handled the same way, so while we talk about storing here, we could also be talking about renaming, deleting or getting a list of files.

We want to send our files directly to S3 from our browser, however we want to have some level of control over who, how much and where they upload to our S3 bucket.

We could leave S3 open to public, but in doing so we are ignoring security. Anyone could upload whatever they like, wherever they like in our Bucket.

So instead, let’s use the power of the VueJS and AWS Javascript libraries to upload a user selected file into a predetermined location in our bucket. Because this needs to look nice to the end user, we will use the excellent Vue File Agent component which creates previews for each file type and includes support for drag and drop, validations and progress bar indicators.

The secret sauce to authenticating and authorizing uploads will be AWS Cognito. The monster just grew another head.

The user will drag a file into our upload window and click an upload button (we could automate this step and not use the button). When the upload button is pressed, we will first retrieve an identity from the AWS Cognito identity pool, and then use the AWS S3 JavaScript libraries to upload the file to our persistent storage. We will attach a listener to that request that will allow us to update the file upload progress bar for each file we are uploading.

So lets’ get started with AWS. Start by signing into the AWS Management Console.

Set up the S3 Bucket.

Setting up an S3 bucket is really simple first go into your S3 Service Console

1. From the Amazon S3 console dashboard, choose Create Bucket.
2. Give the bucket a name, choose a region where you want to store the data. You need to consider data costs vs how close the data is to your consumers. In my case, we are in New Zealand so the nearest data center is in Sydney, however the costs of storage and bandwidth in Australia is significantly more than in the USA when you are talking about the volumes we use, so we generally opt for storing on the US west coast. We also use CloudFront which allows you to cache closer to your end users based upon region, so that may come into play as well… the choice is yours.
3. Check the box that says Block all public access as we will be wanting to limit access to certain IAM (Identity and Access Management) users.
4. Choose your settings for Bucket Versioning, Tags (if you use them for billing etc).
5. Enable Default Encryption using the Amazon S3 Key (SSE-S3) option.
6. Choose Create Bucket
7. Your storage bucket will be created. Go into your bucket, click on Properties and set up the Cross-origin resource sharing (CORS) policy replacing the allowed origins with the URLs of the website(s) where you will be making requests from, in (our case we have 4 ; local development, development, testing and production) as follows:

8. Lastly copy the bucket Amazon Resource Name (ARN) and Amazon S3 website endpoint. You will need this when setting up your IAM policies and Cloudfront distributions.

Set up the Cognito identity pool.

AWS Cognito identity pools support both authenticated and unauthenticated users. Unauthenticated users receive access to your AWS resources even if they aren’t logged in which is useful to display content to users before they log in. Each unauthenticated user has a unique identity in the identity pool, even though they haven’t been individually logged in and authenticated. In our case, we’re going to use that for uploading media.

Go ahead and open the Amazon Cognito Service Console.

1. Choose Manage Identity Pools and then Create new identity pool.
2. Give the pool a name.
3. To enable unauthenticated identities select Enable access to unauthenticated identities from the Unauthenticated identities collapsible section.
4. Choose Create Pool.
5. You will be prompted for access to your AWS resources. Choose Allow to create the two default roles associated with your identity pool — one for unauthenticated users and one for authenticated users. These default roles provide your identity pool access to Amazon Cognito Sync. You can modify the roles associated with your identity pool in the IAM console.

Congratulations your unauthenticated Cognito pool is ready

Set up the IAM

We need to go and add access rights to the Roles created by the Cognito identity pool. This is done by creating an IAM policy and attaching it to the Cognito IAM roles.

Create the IAM Policy

1. Open the AWS IAM console and select Policies.
2. Choose Create Policy > JSON and insert the following policy replacing <AWS ARN> with the ARN of your S3 bucket you fetched in step 7 of creating your AWS S3 bucket. This policy allows the client to Put (upload), Get (fetch) and Delete media as well as listing the contents of the bucket and handling multipart uploads.

3. Click the Next: Tags button. You can assign tags should you wish to.

4. Click the Next: Review.

5. Give the policy a name and description and click Create policy.

Update the IAM Unauth Role

1. Back in the AWS IAM console, select Roles.
2. Search for “cognito” and you should find two roles named something like Cognito_<identity_pool_name>Auth_role and Cognito_<identity_pool_name>Unauth_role.
3. Choose the Cognito_<identity_pool_name>Unauth_role > Attach policies and attach the IAM policy you created previously.

Set up CloudFront

CloudFront is a Content Delivery Network. When someone requests a file from your website, Cloudfront automatically redirects the request to download a copy at the nearest edge location. Because the content is closer, it means that the client is less likely to run into network issues and performance is better. Configuring this is a process of ticking all of the right boxes. First open the CloudFront console and choose Create Distribution.

1. On the Select a delivery method for your content page, under Web, choose Get Started.
2. On the Create Distribution page, in the Origin Settings section, for Origin Domain Name, enter the Amazon S3 website endpoint for your bucket you fetched in step 7 of creating your AWS S3 bucket.
3. CloudFront fills in the Origin ID for you.
4. Leave the Origin Path blank and Enable Origin Shield set to no.
5. Set Restrict Bucket Access to yes.
6. For Origin Access Identity choose Use an Existing Identity and in the Your Identities dropdown select the one labelled access-identity-<your S3 ARN>-<location>.s3.amazonaws.com.
7. Set Grant Read Permissions on Bucket select No.

Next, under Default Cache Behavior Settings

1. Set Viewer Protocol Policy to Redirect HTTP to HTTP.
2. Allowed HTTP Methods set to GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE.
3. Next to Cache Policy click the button “Create a New Policy”. Give the new policy a name and description and under Cache Key Contents choose Whitelist, add the headers Origin, Access-Control-Request-Method and Access-Control-Request-Headers and check the boxes under Compression Support for Gzip and Brotli (note that this is basically the same as the default policy Managed-CachingOptimized but with Gzip and Brotli enabled).
4. Now go back to your create distribution and set the Cache Policy to the new policy you just created.
5. For Origin Request Policy select Managed-CORS-S3Origin.
6. Set Compress Objects Automatically to Yes.
7. With the default settings for Viewer Protocol Policy, you can use HTTPS for your static website.

Next we have Distribution Settings. You may choose to set these according to your specific requirement. For instance if your customers are only in the US, then you are going to get best price/performance by checking Use Only U.S., Canada and Europe.

1. Set the Price Class.
2. I chose to use the Default CloudFront Certificate (*.cloudfront.net). I don’t think that purchasing your own certificate is necessary, but if you are using your own digital certificates, Set Alternate Domain Names (CNAMEs) to the root domain and www subdomain. Important — Before you perform this step, note the requirements for using alternate domain names, in particular the need for a valid SSL/TLS certificate. For SSL Certificate, choose Custom SSL Certificate, and choose the custom certificate that covers the domain and subdomain names. For more information, see SSL Certificate in the Amazon CloudFront Developer Guide.
3. In Default Root Object, enter the name of your index document, for example, index.html.
4. Set the Supported HTTP Versions to HTTP/2, HTTP1.1, HTTP/1.0
5. Ensure that Enable IPv6 is ticked, add a Comment for the distribution and set the Distribution State as Enabled.
6. Choose Create Distribution.
7. To see the status of the distribution, find the distribution in the console and check the Status column.
8. A status of InProgress indicates that the distribution is not yet fully deployed.
9. After your distribution is deployed, you can reference your content with the new CloudFront domain name.
10. Record the value of Domain Name shown in the CloudFront console, for example, dj4p12rmn1ubz.cloudfront.net.

That’s it for this article, your AWS services are all set up and ready to go. In our next article we’ll go into how we use VueJS and the AWS JS SDK to start using these resources in our project.