Blog > Transition to Amazon Web Services for File Storage and Transcoding

Transition to Amazon Web Services for File Storage and Transcoding

Jun 4, 2013

 

 

Since the introduction of Screening Room, we've seen an exciting but significant increase in the number and size of files uploaded to Shotgun. As a result, we decided to investigate our options to upgrade our capabilities for scalable, durable, secure and cost-effective file storage. In the end, we decided that implementing a new storage solution backed by Amazon Web Services (AWS) would provide the most benefits to our clients. Here are some highlights:
  • Amazon's data centers are in compliance with several recognized security standards, including a "demonstrated alignment with the MPAA best practices and AWS infrastructure is compliant with all applicable MPAA infrastructure controls".  (https://aws.amazon.com/compliance/)
  • Amazon S3 supports "at rest" encryption of files, which provides additional security.  (http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html)
  • Amazon can provide higher durability and availability guarantees than our present data center. Amazon S3 is designed for 99.999999999% durability and 99.99% availability of files over a given year... and it can withstand the loss of two of its facilities.
  • By using Amazon EC2 for transcoding, we will be able to elastically scale the number of available transcoding servers to keep up with demand. This will make transcoded media available faster than we can currently with two dedicated transcoding servers.
We'll be starting the transition in the next week and we hope to be 100% over to AWS by the end of 2013. For most clients, this transition will be seamless. If however, you upload files to Shotgun via the API, some small changes will be required before the end of the year. We'll be supporting the API transition on our dev list, so please join us there if you're not already a member: https://groups.google.com/a/shotgunsoftware.com/forum/#!forum/shotgun-dev.

Please feel free to get in touch through support if you have any questions or concerns!

Transition Plan

Phase 1 (June 2013)
  • Move existing uploaded files from our data center to Amazon S3, starting with the oldest files first. 
  • Once we've moved the bulk of the existing files, we'll start transferring all newly uploaded files to Amazon S3 as quickly as possible after they are uploaded. 
  • Transcoding will still happen in our data center. 
Phase 2 (September 2013)
  • New sites and any existing sites that we detect do not upload via the API will be configured to upload directly to Amazon S3. 
  • Remaining sites will continue to use the file shipping process established in Phase 1. 
  • Transcoding will move to Amazon EC2. 
Phase 3 (January 6th 2014)
  • All remaining sites will be configured to upload directly to Amazon S3.