Video Storage/Archive/Backup Solution

[Darren Burns]

We need to setup a video storage/archive/backup solution for our growing video library and am wondering how others do this and if you have suggestions on possible solutions. We work with a non-profit so the budget is not bottomless but we need something in place that is going to work long-term.

Currently there are two of us who need to access the library. One on a PC and one on a Mac (and no, neither of us will change ;) ). The library is on one of the original Drobo's, connected to my PC with network shares (far from ideal) loaded with 4 2TB drives. We are backing up to three 2TB external hard drives. So a single backup and nothing offsite. Reusing equipment we have would be an advantage but not essential.

Workflow is we copy from the library to our local computers and edit there. Resulting files and projects are copied back to the drobo and the local copies deleted. I have a second 500GB hard drive in my PC and we are considering getting an external FW800 drive for my Mac friend to use.

After some research these are some of the ideas I have come across that may contribute to a final solution. Put a small gigabit switch between us, buy a DroboShare and connect the Drobo to the switch. Understandably this will be limited by the Drobo's USB connection. Alternatively upgrade to a Drobo FS which has gigabit ethernet built-in. I imagine there are many other solutions other than the Drobo but I like being able to use any size drive together.

For archiving old footage when the library fills up (which isn't far off) there are two possible solutions I have come across. Burn to blu-ray disks or get a hard drive dock and archive to hard drives. Either way we would be doing 2 copies so one would go off-site.

We would appreciate any advice and suggestions. How are you managing all this?

Many Thanks!

[Roger Wilson]

+1 on the Drobo devices. They are easy to setup, expand and use. The customer support is also top notch.

For archive, I'm using Jungle Disk to backup to Amazon's S3 storage. This gives me an off site backup and can be scheduled to do its work in the wee hours of the morning when I'm not using the Internet connection. I backup active projects and archive completed projects to the "cloud". This gives me the peace of mind if the laptop is stolen or something happens at the home office.

This route is more expensive than archiving to individual hard drives, but Amazon distributes the data over several hard drives, so I'm protected if/when their hard drives fail.

[Steve Kalle]

Ideally, you want to use iSCSI, which one of the Drobo versions has, but I don't know how robust it is. If you search my name and "iSCSI", you will find a lengthy thread where I help someone with their options and explain what iSCSI is and does. Here is a brief explanation: iSCSI is used for SANs, uses gigabit ethernet and is similar to fibre networks, ableit at a much lower cost and reduced speeds. The optimum iSCSI setup includes 2 gigabit ethernet ports on each computer, gigabit switch and 2-4 gigabit ports on the storage device. Using 2 gigabit ports on each computer allows you to "Bond" the 2 ports increasing speed/bandwidth. Furthermore, iSCSI appears as a local drive to each computer rather than network shares (it appears you already know the negatives to network shares). For certain programs, the 'local drive' is more reliable and provides faster access. With iSCSI, you won't need to download the video files anymore because you can edit directly from the storage device.

For good iSCSI devices, Thecus provides several affordable and high performance options.

I personally don't like the Drobo for a few reasons: 1) uses software Raid rather than hardware Raid 2) has FAR slower transfer and random access speeds due to mixing of different drives and/or software Raid 3) 100% propietary, so if the Drobo unit dies, you must get another Drobo if you want to access your data. Whereas with a hardware Raid unit, most parts can be replaced easily 4) due to software Raid, the rebuild times due to a dead drive will be far longer than hardware raid (eg, with 8TB of data, a hardware raid rebuild can take up to 2 days)

For a single person editing, the Drobo wouldnt be a bad idea. But for 2 people editing, the lack of its performance can cause issues, especially when both editors are accessing data simultaneously.

EDIT: I just read that the DroboPro's iSCSI can only be used with one target/computer.

For true long term archival, the only option is LTO tape. Hard drives and Blu Ray/DVD are not designed to last very long. A used hard drive sitting on a shelf for 6+ months starts to rot (there are technical terms for this).

I looked into Amazon's S3 service, but for large amounts of video, it would be more expensive than LTO on an annual basis.

[Perrone Ford]

Steve that's money right there. I am archiving to BluRay short term. Hopefully, a more legit option presents itself soon, or I'll be rotating BluRays annually. LTO is just not a viable option for me at this time.

[Taky Cheung]

I have a Drobo.. and I still have. It's a great idea as long as everything works. You would think your data is safe since having two drives died simultaneously is very unlikely.

And one day you will find your Drobo stop behaving. You will be lucky if you are still within the first year warranty service. After that, if you don't pay for Drobo Care annually, things will be gone.. all gone.

Your Drobo grows bigger and bigger.. All your files are there now. It gets so big that you won't be able to back things up.

So now this is my solution for the growing data storage need
Trayless Removable Harddrive System | L.A. Color Blog

My Drobo is still in use. But I have to use additional harddrive to back up things in the Drobo. For the first year, I have 3 drobo disasters. Many wedding footage are lost.

[Jamie Wonnacott]

I've made some discoveries recently that have helped to address the issues of:
Storage of video files
Backing up projects
Archiving projects, and
Sharing projects across multiple computers

YouTube (Short URL)

I made the video above to explain my findings in the hope it helps others. You can also find more details and links in the article on the New Vision Media Blog: Tips for Filmmakers | Video Storage, NAS & Bittorrent Sync

Hope this helps.

[Jim Andrada]

+1 on LTO. Generation 7 is within a few months of shipping with 6.4TB uncompressed per cartridge.

If you're going to store hard drives on the shelf you WILL encounter "bit rot" sooner or later. As long as hard drives are running in an array the system constantly runs checks on the data and repairs errors. Take the disk offline and nothing monitors or repairs it.

No user writable optical medium is truly archivable.

[Andrew Smith]

Spinrite is great for maintaining HDDs against bit rot / magnetic flipping.

See https://www.grc.com/sr/spinrite.htm

Andrew

[Jim Andrada]

Exactly - you have to mount the disk and power it up to run Spinrite - as long as the drive is just sitting on the shelf it is going to have problems that will be undetected.

If you want to store HDD's you should (must???) periodically power them up and run something like this to check them.

People are still using tapes that have been on the shelf for 40 years. Tape cartridges are designed to sit on the shelf for years. HDD's are not. Tape error correction encoding takes into consideration the ways in which a tape can be physically damaged - for example scratched longitudinally. So the data are scattered in such a way that the data and the associated ECC codes are never stored in the same straight line.