In 5 years storing video on hard drives, I've lost some data with all solutions except RAID1.

RAID5 seemed particularly shaky, despite the promise of savings.

Go simple. RAID1 = instant data backup onto 2 drives inside the array. You'll need exactly twice the capacity, so for 1TB storage you'll need two 1TB drives. (Does not seem like a deal breaker at current low HDD prices.) And then you're all set. Even if the RAID controller card (Drobo and WD Books come to mind, as well as internal PC cards) fails, you still have your data intact on at least one of the two RAID1 drives.

Just few months ago we have went for tapeless acquisition, moving from reaally old now Betacam SP - somehow we've managed to skip the DV revolution in terms of acuisition format, so our equipemnt did pay for itself numerous times. this should show that we are not willing to spend money without justification.

This presented us with exactly the same problem. And this is what we've come up with.

While LTO drives are still pretty expensive, we went for considerably cheaper DLT drives. We use Quantum DLT-V4 drive, and while it is slower and much lower capacity than what George Kroonder stated about the Ultrium drive, it still stores 160GB native, and life expectancy beats any dvd, or presumably bluray bluray being a new format, as against to the DLT technology which has been around for almost "ages"

We have it hooked up to the RAID5 server where we store original files from EX and other sensitive data.The drive has e-sata and usb connection, and any XP machine has a backup software to use it with. So it really is a no brainer, considering you handle your project files in a coherent manner. Gather all files in one folder, mark it for backup, put in the tape, and for next 30 years or so you should sleep well.

We charge client for archiving on the per-GB basis. And most our projects easily fit within the 160GB tape limit.

I know this is not a perfect solution, but maybe someone will get inspired as we were inspired by Bart Walczak - dvinfo user and a friend of ours :)

It's a mystery why people gravitate to a product like Drobo. DIY RAID enclosures and arrays have been around for years. One of the best, IMHO, is available at Addonics Technologies - Making Data Storage Simple, uses SATAII (3Gbps) hard drives. But, for all the reasons cited above, RAID is not a good solution for long term archival.

As to RAID5 vs. RAID 1 - we've built our raid 5 some time ago, and now, with the dropping prices of drives, we are considering changing to the simple raid 1, as it is just faster solution than raid 5 - especially when saving files.

Drobo vs RAID 1 vs Drives on the Shelf vs Optical
 

Here's my reasoning:

Shelf: A good solution with the advent of cheap, reliable drives, but with risks. When on the shelf, they are not monitored. Drives lose magnetism over time, resulting in data loss. Drives can also 'seize,' it's called stiction (I know, I worked for one of the early Mac external drive companies and there were a lot of failures due to various drive hardware problems). The Mean Time Between Failures (MTBF) is just that. It's an average of the lifetime of a drive over thousands of data points. In one case, a drive could simply break in 1 day, another case could be viable for 20 years.

Large Capacity Optical: When Blue-Ray disks come down in price, this may be the best solution. However, many people don't know that Optical media is not necessarily a permanent solution. Optical disk substrates are created with aluminum, which, unless perfectly enclosed by the polycarbonate around it, can corrode. If you try a CD you've burned 10 years ago, there is a good chance there are some unreadable areas. If that area is an important piece of video, it's lost forever. Higher quality media can help mitigate this with more inert materials and coatings (gold, etc) but are a lot more expensive.

However, in addition to whatever 'online' solution I have (below), I always have an 'on the shelf' backup of the data as well.

Raid 1 (mirrored drives): I have a number of RAID enclosures that are now configured as RAID 0 for speed. The enclosures are not that expensive, perhaps $150 for a low end solution like from OWC computing (plus the drives). I could also configure them as RAID 1 as a backup if I chose to. When you configure a RAID to be 'mirrored' you lose the speed benefits of RAID 0. Any good online RAID 1 or other redundant solution, needs to be powered and monitored in some way for drive 'health.' In the case of a drive failure, there are benefits of being able to use the good remaining drive or put it into another enclosure and utilize it immediately if the enclosure goes down.

Raid 5, 1+0, or Drobo:

Raid 5 is a solution similar to a Drobo, which allows for the failure of 1 of 4 drives without data loss. It usually requires expensive enclosures and usually associated monitoring software. The drives need to be matched and the same capacity. The speed is about that of the slowest drive in the array.

Raid 1+0 is a set of at least 4 drives which are mirrored and striped together. It has the speed benefits of RAID 0 with the safety of RAID 1. It's complex but it would suitable for your 'working' drive setup. I'm considering standardizing on this solution in my Mac Pro, but I'm still researching it to see if others have had good luck with it. It's quite a commitment. And the capacity is limited and not expandable and you'll still need an additional archival system.

Drobo: Very simple, easy to use (basically plug it in and it works) elegant software (it can be configured to email you status about the drive including failures) and is expandable by replacing the drives with higher capacity drives. It's not that fast. Maybe 30MB/second over Firewire 800. I don't it as my 'working' drive, though I could in a pinch. I would recommend the Drobo for the same reason I recommend Mac computers, it's simple and it just works. You don't have to worry about configuring it or having much knowledge about RAID, and it's relatively inexpensive for what you get. The fact is I have and could have setup any type of archival system, but I have settled on the Drobo for the reasons I've stated. Maybe it's a mistake, time will tell.

Whatever solution you use, I'd recommend a 'shelf' backup (optical, data tape, hard drives) but I'd also have an online solution as well. And I'd have a regimen to put the 'shelf' media online periodically to ensure it's not degraded, and if it has, back it up again.

This is all a pain, but it's the digital age we live it. We're producing content exponentially, and we have to figure out a reasonable way to safely store it.

I own a WD Book of 2TB capacity, and it has 2 built-in drives (supposedly user-replaceable), and an internal controller that can be configured as either Raid0 or Raid1 (with 1TB total capacity for latter.)

It has USB, FW, and eSata connectivity, and is very small in size.

Comparing to Drobo, what would be disadvantages of this solution, if any?

I wonder, Keith, why you feel the need for an online solution at all? Isn't the magnetism loss over time the same whether you're using the drives or not? I would think that except for the time that you're adding files to the drive, isn't it better to leave it alone? As for blu-ray video decay, is the risk the same if you're just archiving your BPAV files? It's not a video blu-ray, just a data one. As for a raid array, I'm thinking it would be just as easy to get a couple of raid drives, one backing up the other. That gives you the performance for editing of Raid 0, and the redundancy of raid 1 mirroring. Once I run out of space, just get another pair.

Here's my archival/work storage thinking so far:

Get a couple of mini raid drives for location; that improves copying time by having a couple raid drives in one enclosure, and peace of mind having the second unit mirrored. Minis keep the weight down.

Do the same for editing, except they don't have to be minis. When they're full, get another pair.

Dump the minis onto blu-ray to back up dailies and onto a single (shelved) hard drive as well, just to be safe (& neurotic--that way two different storage media need to fail in two different ways. ;) . Maybe the blu-ray (as opposed to just HDD) is giving me comfort in a retro way: it will be nice to actually have something real to pick up off the shelf, like a tape! I also like that it's discrete: the first shoot on the first half-dozen (or so) disks, the next shoot on the next half-dozen, etc.

The magnetism loss issue probably does happen with an online drive, although online drives do periodically check data loss and rewrite a marginal sector if found. The act of reading data on a drive initiates this. Also an online solution such as a Drobo will alert you (via email or other means) if the drive goes bad. A drive on the shelf can't alert you.

Blue Ray: It's not video decay, it's physical aluminum substrate on an optical disc that have a chance of becoming unreadable over the very long term (5 years or more). This is an industry secret but you can look it up. The truth is most affordable optical storage isn't 100% reliable over time. It, however is probably on of the better 'shelf' solutions available, as it's not susceptible to mechanical failure or erasure from a magnetic field.

No solution is going to be 100%, its a matter of economy vs convenience vs safety. It seems some people think an offline drive has a better chance of staying intact than an online because an online is just used more, reducing it's effective life. I suggest that an online solution can alert you if things go bad and is safer. I don't say throw away the offline archive solution, just realize it's not 100% reliable. If I had unlimited resources and a full time tech to monitor it I'd back everything up to Blue Ray Gold disks, plus have RAID 1+0 s that have health monitoring on all the time and periodically run a script the does a read of every bit of data on these drives. Mike, this is similar to what you have, and I like it. For me, right now, this isn't practical so I currently have a bunch of Raid 0 drives backed up to on the shelf drives but I'm transitioning to Drobos backed up to on the shelf drives. My online work stays on Raid 0 for speed. The inherent (theoretical) safety of the Drobo solution means I can be a bit lax on my off-the shelf backups.

I'm not pretending to be the be-all end all authority on this issue, but through a couple of decades of experience and experiencing the pain of data loss, when it happens you wish you had archived in the best way possible.

Part of these discussions is to learn new things and perhaps better ways of doing things. I'm open to better solutions that offer the a better combination of speed, safety, ease of use and economy so please keep the comments, challenges and questions coming!

Keith, question: what kind of HD video do you work with?

I use Cineform Prospect HD at High, and I'm able to edit up to 4-5 streams off of my RAID1 system.

Hi Alex

I think it's a pretty good deal if you just need 1 TB and a small, self-contained solution. I considered inexpensive solutions like this.

Here's a small analysis of the cost: you'd need 3 of these for a cost of about $1,000 to equal one 4TB Drobo. Drobo is about $500, 4 x 1 TB Drives another $600. Total $1,100.

That's 930GB once formatted and you'll run out of space pretty quickly archiving the EX1. It's not that fast in mirrored mode (but neither is the Drobo), replacing a drive if it does go bad is not as easy as the Drobo, I don't know about what monitoring software it has if a drive goes bad... will it flash red or something, what if you have it in a closet, will it email you if the drive goes bad?

It's for these reasons that I decided on the Drobo solution as a longer term solution. I was getting tired of buying small RAID drives and piling them up one after another. But if your storage needs are modest I don't necessarily think it's bad. You may want to buy 2 of these, keep them in Raid 0, then you have the benefits of a large capacity, Firewire 800 / eSata-worthy speed, and a good backup in case one goes bad. It's not a mirrored solution so it's only backed up as of your last daily backup, but not terrible.

The thing that's also kind of appealing about the Drobo is you upgrade the drives pretty easily. As drives become high capacity (I have 4 x 1TB drives in mine, but 1.5 TB drives are becoming more common and affordable) you can take out the old drives, use them for 'shelf' archive, and pop in a newer higher capacity drive. I anticipate that will happen to me about once a year or so.

I hope I don't seem like a shill for Drobo. I deliberated a bit before getting one, and had a couple little issues with it when I first got it, but I just like the idea, the smartness of the software, and the simplicity.

Maybe in 5 years hard drives will be relegated to the museum and we'll have flash or other types of cheap, stable archival media that will make all this discussion moot.

Alex, I edit with HDV, Prores 422 or XDCAM ex. Much lower bandwidth requirements than your Cineform I believe (25, 35, 145 mbit/second). I usually work online with my RAID 0 inside my Mac Pro. I keep other stuff on a variety of RAID 0 drives on a Gigabit ethernet NAS. The NAS can get me 70 MB/sec transfer rates. So I'm a Mac and you're a PC I assume?

I wouldn't use the Drobo for online video editing unless in a pinch, if that's what you're asking. I've clocked about 20- 30MB second, which theoretically would work but would result in some random delays, I think. Haven't actually done much real editing like that though.

No, actually I was just wondering why do you need RAID0 for online edits at all.

Last time I needed RAID0 was when I was capturing *uncompressed* HD 1920x1080 live.

With Cineform, there's simply no need for RAID0 from speed viewpoint.

So my production system (yes, a PC - will you make me into a stereotype now? ;) has internal RAID10 (essentially RAID1 x 2, for double the capacity) and works fine.

I think controller card is extremely important, so I had to shell out for Areca (after not so good experience with others), and am very happy with it so far.

Of course all my PCs are powered via UPSs (uninterruptible power supplies), so power loss/brownouts/blackouts/spikes so not affect my data.

I use RAID 0 not because I need it necessarily but because it's fast. I guess I like to have the fastest available system for online editing. I do a lot of disk intensive work, effects, filters and such, and the RAID 0 seems to make a difference. Although SATA is theoretically fast, the disk is the bottleneck. RAID 0 makes throughput faster. Maybe that's unwise.

Sounds like you have a great reliable system, even though it's a PC :)

I think this thread has gotten off track.

Thanks, Keith, for your reasoning.

Alex, editing in FCP with XDCam footage the speed difference is not insignificant. I did a comparison with a raid 0 (two drives in one enclosure) and non-raid drive and was getting about a 30% increase in speed. Transferring a 13.5 minute clip via XDCAM transfer (that's the program's default for keeping files under 4gb) took 4 mins with the non-raid and 2.75 with it.

Don't think the thread's gotten off-track, Jay, as it's very hard to separate out one's online system with one's archiving system, since in many cases they overlap.

In my case they're the same :) *

As for the speed, yes, I actually use RAID10 which is two RAID1's coupled into RAID0. Makes it faster than just RAID1.

I agree that of course file transfers are much faster on RAID0-based systems.

------------------------
* Well, almost. I have two 2TB RAID10 arrays... one production, which I use for current projects. Another to dump the *finished* project on for archiving. This way archiving system does not get taxed as much as production one by constant HDD use, and I feel this is safer for storage.