Adventures in Data Storage, Part 1

Back posting again after a holiday weekend here in the USA as well as an adventure to recover damaged data. On Saturday morning, I wanted to check up on a hard crash that had occurred on the main “Frankenserver” that stores a backup of all the Outside In and SV2 Studios files.

A Frankenserver being upgraded in 2011

Here’s a pic of secondary Frankenserver being upgraded last year – I call them that as they are built from the parts of other machines to recycle parts and keep the film’s costs as low as possible.

One question I get a lot is how I store and backup data. I’ve made a simple diagram that gives an overview of the design and process.

Dataflow in the film's storage and backup process

This diagram is simplified but all the key steps are there. I have two primary workstations – the primary is the fastest, most powerful and has a large, fast disk array for photographic storage and the secondary is used when the primary is tied up and also for freelance work as needed.

I have 3 other workstations – an audio/recording workstation and secondary render box plus two low-end boxes: a Linux machine and a Hackintosh. Then there are two Frankenservers, the primary that has 27 TB of storage and is connected to a cloud backup and the secondary provides additional utility storage (workstation backup etc.).

The primary Frankenserver is connected to cloud backup storage (CrashPlan). The overall design of this is to keep the costs of the film 1/10th of what it would normally but create a system that can recover, eventually, from even a catastrophic data loss – say an alien warship destroying my house (but not destroying earth, but then again, that will not be a big worry at that point).

This design also allows me to work on the film on multiple stations although the primary workstation is fastest for the heavy lifting. It consists of low-cost, DIY, home-built computers overclocked to provide near professional workstation speeds at 1/4 of the cost. And the storage and backup costs are a tiny fraction since it relies on the very cheapest hard drives and cloud storage that costs only $5 a month combined with a $20 month upgrade on internet service for fast upload speeds.

The bottom line is all critical files are backed up in at least two locations and 3 to 6 copies of each file. Storage costs on the film are currently 4 cents a gigabyte and offline backup costs are less than 1 cent a gigabyte. Currently I have about 6 Terabytes backed up offline but the number grows everyday.

It’s a good design especially given the cost – the only disadvantage is a slow restore process in the event of large data loss but this past weekend, it was tested.

PART TWO TOMORROW – what happens when the main server motherboard dies and corrupts the main data storage array.


Comments

Adventures in Data Storage, Part 1 — 1 Comment

  1. Pingback: Adventures in Data Storage, Part 2 | OUTSIDE IN

Leave a Reply

Your email address will not be published. Required fields are marked *