Mission-Critical Video storage solutions
 

Another two of my external Lacie hard drives just started giving
signs of slow demise. One of them - the 300GB one shows up in "My
Computer" after several attempts to connect, and that's the one that
clicks. The other one, 500GB, is only visible in "Disk Management"
under Administrative tools of the Windows XP OS. I am going to start
finding ways to recover the data, the 300GB one contains pictures of
my niece since she was born in 1998. I have discovered that I would
never trust Maxtor ever in my life, and that's the kind of drives
sitting inside the "sexy" Lacie enclosure of the 500GB one. I kept
all the important data on the 300GB, which is a Seagate inside, yet,
I am now facing this painful situation.

Seems like four years is the point when one has to trasfer all data
to a newer storage option. Do hard drive manufacturers do it on
purpose so people go out and buy new ones? I know for a fact that
cell phone manufacturers use this horrible tactic.

What are your experiences with mainstream hard drive manufacturers
when it comes to hard drive failures? Do you have solutions to
recover data and, perhaps, finding mission-critical storage for
contect like raw video?

I have created a yahoo group dedictated solely to data archiving. If you ever lost valuable precious data to a failed hard drive or scrached optical media (CD/DVD/Blu-Ray), leaving you wondering if there is a long-term future-proof data archival solution out there that will not break the bank, this group is for you. I cordinally invite you to join to share information and find solutions to be able to preserve the data for the future generations.
Here is the group;
http://tech.groups.yahoo.com/group/archivist/

Thanks

Renat,

1/ You don't say which models you are having trouble with.
2/ You do know that Seagate bought Maxtor, don't you?
3/ I have used Maxtor drives in the past. Never had a problem.
They got replaced when time made them unacceptably small & slow.
Like the computer they were sitting inside.

4/ I hang all my computer/video/audio equiptment off of a UPS.
5/ It appears from LaCie's website that most of their designs are not only
fanless, but totally enclosed, allowing no air circulation. I would avoid
designs like these. Like the plague. Regardless of the marketing,
it is an inadequate design.

The only hard drives that can get away with this are generally the smaller
Western Digital Passport style hard drives, where the whole thing is
operated and powered solely by a USB cable. That ensures power
consumption is 2.5 watts or less and therefore there just isn't much
heat to get rid of.

I tend to be particular about making sure my drives stay cool.
Heat is a big enemy of hard drives. I happen to use Western Digital
MyBook USB Drives in 320GB and 500GB capacities. They are open
enough to allow some air circulation. In addition, when in use
I have them sitting on a laptop cooler actively blowing air through
them.

Noisy, but makes a big difference in the hard drive operating temperature.
If I forget and add/turn on the laptop cooler after the hard drives have
been running for a while (hours), you can really feel the heat pour out of
the drive enclosures.

My desktop computer uses an Antec P180 case. All the hard drives
have a nice big 120mm fan sitting right there blowing air over them.
Again, makes a big difference in the operating temps.

You do know hard drives aren't supposed to get over 100F/40C, correct?

Thanks Bill!

I do agree with you that hard drives must have sufficient cooling. But do you think that will prolong their 4-years lifespan? I have built systems with Antec P180 cases. Nice internal design, albeit hard to access the internal hard drives once everything is set and go.

To answer the rest of the questions you posted;

1. The drives are:
500GB Lacie d2 Big Disk
300GB Lacie d2 Big Disk

2. Yeah I am aware that Maxtor owns Segate, it's been sometime now.

3. I always had problems with Maxtor drives, I teke off my hat to them though for not messing with the Segate production after aquisition.

4. I am not sure if UPS can improve the lifespan of hard drives.

5. Lacie has been on my shitlist since the first 500GB drive failed back in 2006 after I came back from a vacation and powered up the workstaion. This company is the worst in the history of computing! I hope they never existed and it was all just a nightmare. All they do is outsource the internal components, sometimes making a "board" to raid two Maxtor drives, and make packaging look to resemble that of Apple's. I have most of the people I talked with about them back me up since they trusted them when they first emerged.

Now, what a way to measure the temperature of a hard drive to stay within 100F/40C?

Thanks again!

Yes.

As I suspected. Totally enclosed design, no airflow.
Replace as soon as possible. They are just waiting to die.

Actually, it is the other way around. Seagate bought Maxtor.
Seagate & Western Digital have been the two biggest drive manufacturers
for...the past 15 or twenty years. There used to be a Quantum, but they
got bought by Maxtor. And now Maxtor has disappeared into Seagate.

Back in the 1992 - 1995 I lived in Michigan outside of Ann Arbor.
I had constant computer failure problems. Hard drives, tape drives,
phone answering machines....

I was wiping my hard drive ~every two weeks and reinjecting my system
from the last tape backup. Then I bought my first UPS and all of that went
away.

A UPS ensures clean power at all times. I am not going back.

Most companies outsource production of the main components. There
just aren't that many companies that can afford the multi-billion dollar
investment for the factories needed to do mass volume super precision
component production. So look for the majors, because that's where the
parts are coming from anyhow.

I haven't figured it out for external USB drives, but if it hangs off of a
PATA or SATA connection, this program works pretty well.

http://www.filehippo.com/download_speedfan/

I am interested in your thoughts on Lacie having read your bad experience. i just bought the 500 gig LaCie de Quadra Hard drive, which is possibly an improvement over the one you have? it is touted as having the lates heat sink technology..whatever that means.

bill, bothell wa

All hard drives are subject to failure.

For anything long-term I use a "mirrored RAID" consisting of two drives. Both drives are written to simultaneously. In case one drive goes bad, the other one continues to run.

The important issue is discovering when one drive goes bad, and knowing which drive is still good.

Apple's Disk Utility isn't much help in this case.

However, something called SoftRaid might be. I have to check it out and see how well it works. It's supposed to provide the user with an immediate warning if a drive has gone bad, and be able to help identify which drive needs to be replaced.

I used to think so, but now I'm not so sure. I just tested 2 aluminum enclosures side by side, with the same model 3.5" 7200RPM drive.

One enclosure was "sealed," and the drive temp stabilized at 95F in a 75F room.

The other enclosure was vented (but fanless). The venting made the drive's spin noise easily 3 times as loud, and the drive temp stabilized at 105F. Maybe it was a matter of poor venting design.

I agree with you in theory, but I've settled on the quiet, sealed enclosure, and it's keeping the drive's temp identical to the one in my desktop chassis. Sure, I'd happier if both of them were running at 85F, but I don't think 95F is a particularly big deal.

I have decided to never buy external drives. Simply buying internal SATAII drives and enhousing them into self-built boxes with 120mm Nexus noiseless fans pointed at the drives. The aim is to achieve almost cool to touch surface of the drives using this approach.

If those fans are really quiet, that sounds like a great idea. Do you have any pictures? Sourcing the fan isn't a big deal, but I'm not much for constructing my own box...

Cool to the touch would be ideal, of course.

Actually the boxes are not really self-built. I just took them off of old Dell cases. You can slide the hard drives in and attach the 120mm Nexus fan on the openning opposite to where you connect all the cables of the drives. The box I am currently using even has some holes on the sides, great for attaching another fan. The cooler the better the lifespan of the drives. Sorry no pix though...

As others have said, all drives will fail. It's not "if" but "when." Maxtor (now Seagate), Western Digital, IBM (now Hitachi), etc.,. have all had times when a batch of bad drives have hit the streets. A mirrored RAID will help get you back up and running quickly if a drive eats it, but if a power spike comes thru and fries your gear the RAID doesn't offer much protection in that situation. I like to back-up footage onto FW drives and keep them disconnected & unplugged for that reason.

For mission-critical storage back up to data tape. Keep one tape on location and another copy in a safe-deposit box or something.


-A

Right on the money, Andrew.

I am about to setup a RAID0 configuration on two 500GB Seagate 7200.11 drives but wonder if I should set it to RAID10 by adding another drive to backup these two on the fly. I have never setup RAID10 and wonder if you'd tell me how it's done. Do I first setup the RAID0 on the two 500GB drives and then turn the resulting 1TB drive to RAID10 with another 1TB? Also, does the backup drive have to be 1TB as well or can it be less, like 500GB?

As far as tape back up; What is the most durable mission-critical brand to go with? I heard there's holographic technology out there now, albeit very expensive for an idependednt video professional (InPhase Tapestry drive runs for $18k and can backup to 50GB disks - each ne costing $180 or so).
Blue-Ray is promising but it's very slow to write and read at this point. The fastest one being 6X (40 mins to write a 25GB disc), and that's when the 6X media will be available. Currently there's only 2X, which takes 2 hours to burn 25GB disc. I tested it.

For those who who haven't heard, I have created a yahoo group related to archiving data for future generations to come. I cordinaly invite anyone who is not careless about preserving data. http://tech.groups.yahoo.com/group/archivist/

cheers

Hi Renat,

The topic you started was "Mission-Critical Video storage solutions", yet the discussion is basically about external "consumer" drives in single disk configuration.

Using a stripe-set (RAID0) with will (more than) [i]halve[/u] the reliability when using two drives as all data is lost whein either one of the drives develop a problem. It gets progressivly worse with more drives added to a RAID0 array.

Using mirrored drives (RAID1) provides you with better protection against hardware failure, but you only "get" halve the capacity and if you get a power surge, both drives can get damaged, especially if they're in the same housing. Newer RAID controllers can mirror an uneven number of drives (i.e. 3, 5, etc.). I personally find this an odd system (no pun intended).

Mirrored drives give you faster read times and slower write times. A stripe set is the fasted configuration. You can combine the twe to RAID10, but you need at least 4 drives and, get halve the capacity, but at the best performance.

RAID5/6 compromise speed and capacity, i.e. you get more capacity then when using RAID1, but you get a little less speed compared to RAID0 (or RAID10). You need at least three disks for a RAID5 array.

With the current PCIe SATAII controllers and disks, the speed of a RAID5 array will likely be more than enough for your needs, especially if you use USB / FW400 external disks now.

If you are indeed are looking for "Mission-Critical Video storage solutions" you should pay attention to the housing, the connection, redundancy and the drives. Basically in that order.

George/

I don't think any experienced IT professional could ever recommend "consumer" optical storage as a long-term archival worthy product. I certainly would not and I've been in this field for almost 15 years. Optical WORM media (whole different game compared to CD's and DVD's) never really took off in the market and certainly did not displace tape as the preferred backup medium or even the preferred archiving medium among smaller and mid-sized businesses.

But if you've got $20K to blow on an archiving solution, then perhaps you should look at the magneto optical stuff. Otherwise, you could spend a fraction of that and go with a quality tape product - LTO, DLT, etc. These are industry standard formats that have proven themselves as appropriate solutions to this problem, a problem that is beginning to hit more and more small businesses due to compliance requirements from regulatory agencies.

Thanks George,
I started this thread with an example of external hard drive failures. However, the RAID conversation is also related to this topic.

Seems that the best way to go around the RAID decision is to simply set the two 500GB drives to RAID0 and then use a program called "SecondCopy" to backup the newly added data everyday to a different drive. This program allows both manula one click backup as well as scheduled. This way the video editing write/read access while video editng is not compromised on that RAID0 1TB.

@ Aaron

LTO, DLT write/read access is much slower than that of optical, isn't it?

Thanks again to everyone!!!

I woudn't even think about DLT (Digital Linear Tape) anymore. It's about as dead as a technology can get and not be buried. It's been almost totally blown away by LTO (Linear Tape Open). Even Quantum (the maker of DLT) recently bought the tape piece of Seagate, which was one of the original memberts of the LTO consortium, together with IBM and HP and is now making LTO their major focus in tape.

At 120 MB per second (native non compressed) I think it is way beyound anything you'll get from an optical drive. The biggest perfomance issue with tape is the time it takes to space down the tape to where you want to start reading. (Time to first Byte)

Current LTO Generation 4 has native capacity of 800 MB per roughly 4" sq cartridge. Generation 5 will be along around the end of the year or early next at 2X the capacity and higher data rates. Generation 6 is spec'ed at 3.2 TB per cartridge (uncompressed)

The IBM/HP/Quantum consortium is committed to developing at least 6 generations of the product and nobody is thinking of stopping there, so it's a very safe bet to be around for a long time yet. Yearly sales for all vendors total over 400,000 units, another indication of why it won't disappear anytime soon.

Drives by IBM, HP, and Quantum are highly reliable. I'd avoid drives made by any other company. Dell OEMs the drives and resells them so they are easily available.

It's a super technology. There are lots of small LTO tape libraries available at reasonable prices (in the ballpark of small pro cameras) that can really help streamline any backup and archive application.

DISCLAIMER

My day job is as the business interface between two companies that have been partnered in developing the LTO family of products. I've been working on the project since the beginning.

I'm sure you meant that to be 800GB per tape, just to nitpick.

George/

George,

Right on - I meant 800GB, I'm sitting in a hotel room in Tokyo and am fumble fingering the in room PC keyboard

Jim,
Excelent info! Thanks so much.
So, how long would it take to write, say, 60GB of data to an LTO gen 4 tape? Does it matter what brand gen 4 drive you have? If choice is among the big guys you mentioned - not OEM's of course.

Also, what do you mean by performance issue in regards to tape vs. optical media. Isn't it that you just insert the tape and then go to windows explorer and start browsing files right away?

Thanks again!

Tape media, including LTO, is cheap (at least per GB) but a bit cumbersome to handle/operate.

I would suggest to look at professional optical disc storage, like the SOny Professional Disc. It is not as fast as tape in write spead, but it stores 23 or 50Gb of data with a shelf life rated at >50 years. And you can just browse it like you do a CD/DVD.

They make a good project archive medium.

LTO has several advantages, but I believe it is less suitable for archiving unless you have enormous projects and a good climate controlled storage facility. Tape is more fragile than optical media especially when it comes to heat and moisture.

George/

Yeah, I checked the prices and was astounded that the cheapest LTO4 drive goes for like $3200. The media is cheap but the drives are ridicuously expensive and huge too. I think Blue-Ray has promise value-wise.

The most impotant thing is to have some kind of formal archive/back up plan and to follow it.

After that, it's all a matter of degree.

I don't know that I'd consider LTO for all applications.

On the other hand, if it's appropriate for your situation, it's hard to beat.

Maybe a rough rule of thumb would be that if the drive is less expensive than your camera or your sound setup, then you should look at it pretty seriously, otherwise optical might be OK. Another criteria might be the volume of stuff that you want to archive, Considering that one LTO cartridge can hold 50 or 60 MiniDV cartridges and the number will go into the hundreds in the next couple of years, I think people should look at it for any kind of high volume usage. Data rate is also a big factor if you're archiving whole projects.

If you're using HD and something like high or film scan level CineForm intermediates - or even uncompressed, then I'd really get serious about considering LTO

Re fragility - not sure I'd consider the media fragile. We re-use cartridges 5,000 times apiece in our drive test line and we've seen some that will be fine after 20,000 load - unload cycles. Normal home environments are climate controlled enough from everything I've seen. (Well, maybe a non air conditioned home in Singapore or Tucson might be stretching it a bit!) We also test the drives at a lot of load unload cycles each before OK-ing them for shipment. Some of the networks in Japan are using LTO as their archive of everything they broadcast.

Cleanilness is impportant - you don't want to get dirt inside the cartridge. On the other hand, there's a spring loaded door that keeps the cartridge closed so that no tape is exposed.

In all, it's quite a different animal from MiniDV tape. Which is all I really wanted to say, as "tape" gets a bad reputation based on characteristics of MiniDV tape and it's important to realize that there's quite a spectrum of "tape"

I built an extra computer that acts strictly as storage. I bought the cheapest Intel CPU and decent motherboard. Rocketraid makes some nice cards that can handle 2-8 drives. I then built a RAID 5 out of 8 - 500 GB drives. Thats just over 3.2 TB of storage. This machine is turned on/off only when I know that I need something off of it. I did have 1 drive die recently, but the computer still booted up, told me of the drive error and it was replaced. I was even able to access the data while the new drive was being re-built.

I used to rely on RAID but one day the RAID controller card in the PC died and trashed both disks. In a real commercial array they will usually have provision for dual controllers.

Even with RAID, there should still be a separate back up and archive process.

Any advice about implementing LTO with a desktop computer? HP have told me they don't support LTO drives with desktops. I have a desktop (core 2 duo) running either XP or Vista. My hard drives are SATA or eSATA. I think LTO-3 may actually outperform my disk drives, and potentially damage the tape drive or tape, as it requires a MINIMUM data transfer of 27Mb/second. LTO-2 might be appropriate for data rate, and has a maximum data storage of 200Gb uncompressed per cartridge; presumably uncompressed is what I would want for video files. The drives are all either SCSI or SAS interface, so I assume I would need a PCI express card to give me the correct interface. I was thinking of the Adaptec 2250300, which claims: "PCIe x1 host connectivity, Compatibility with SCSI and tape hard drives, Ultra320 SCSI single channel.
Total cost in UK about £1,000 for hardware, and tapes cost £20 for 200Gb. Has anyone actual implemented something like this. I don't want to buy the hardware, and then find it doesn't work.
Patrick

Now you're embarrassing me!

I've been working on LTO development (but primarily on the business side) for 10 years or so, but can't keep the various models straight in my head.

As well as "little details" like the best way to actually hook one up to a PC, what application to use for it, etc. All these minor details that are so necessary to actually USE one.

I know a lot about the structure of the roller bearings, what kind of coating we apply to avoid having the metal roller flanges cut by the abrasive edge of the tape, what kind of motors we use where and why, etc, I'm just a bit DULL when it comes to plugging one in and using it!

I have a call in to the head of the LTO drive engineering team at the XYZ Corporation (3 letters, so it isn't HP!) and I'm going to try to get with him in the next day or so and try to address your questions and comments.

Patrick,

One additional comment:

Re compression, the LTO compression algorithm is completely lossless, so it won't mess up your data in any way. I doubt you'll get much of any compression of video, but I"m also pretty confident that LTO compression won't make the files bigger (it can happen in some pathological cases with some algorithms)

Normal coded info (ie text, spreadsheets etc) usually will compress by a factor of 2 to 3 on LTO but everybody just claims 2X.

Sort of in the category of more than you probably want to know, LTO drives have a set of write heads and a separate set of read heads. During write, the data as written to tape is automatically read back and compared. if there is any discrepancy, the write is simply terminated, the drive skips forward a predetermined distance, and does a re-write of the block in question. if the write completes normally, various block end and error correcting data is appended to the block.

On readback, if a block terminates without the appropriate end of block info, the drive knows that the block is bad and throws it away and continues reading down the tape until it picks up the correctly written copy of the block.

All of this makes for a busy set of logic chips - there will be up to a total of 32 (and soon many more) reads/writes going at once. We're hard at work on Gen 5 which will have a native capacity of 1.2 TB per cartridge.

Because of the writes going in parallel, tape can be written at data rates in excess of what a single disk drive can sustain. This is normal and the engineers allowed for it in the design, so I wouldn't worry about damage to the drive.

It's a completely different animal than the helical scan drives used in video applications.

Now if I just knew a few details about how to actually use one...

Hope to have answers for your questions tomorrow or the next day.

Jim,
Thanks for looking into this.

This is what HP says about data throughput, which is what made me nervous:
"The tape drive should be operating at or above its lowest streaming data rate to achieve the best life for the head, mechanism, and tape media. If the data is not sent fast enough, the internal buffer will empty and the drive will not write a continuous stream of data. At that point, the drive will start exhibiting what is called head repositioning. Head repositioning is also known as "shoe shining" and it causes excessive wear on the tape media, on the tape drive heads, and on the mechanical tape drive components."

For LTO-2 Lowest native data rate is 10Mb/s, LTO-3 26Mb/s, LTO-4 40Mb/s.

http://tinyurl.com/593ys2

LTO-2 at 10Mb/s is presumably not a problem with a single SATA drive, but I am not sure it is possible to guarantee 26Mb/s for LTO-3 at all times. LTO-2 tapes store 200Gb uncompressed and cost the same as the LTO-3 tapes which store 400Gb.

Thanks for your help,

Patrick

It's also called back-hitching. The term "shoe shining" comes from the behavior of the drive when incoming data doesn't keep up. When the drive runs out of data it stops, but stopping isn't instantaneous, so it will back up and then read forward slowly to find the end of the previously written data before it starts writing again. The back and forth motion makes people think of the old fashioned way people shined their shoes by rubbing a cloth back and forth across the shoe surface.

Patrick,

I had a short discussion with the head of the LTO development group at XYZ I sent him your posts and he and I are going to have a discussion with his technical guys first part of next week after the technical guys have had a chance to look at them.

His first take on the issue of damage from shoe shining was that it wouldn't cause any stress to the heads or media, but there might be some slight concern re wear on the guiding structures from the edge of the tape itself moving back and forth. He didn't think it would be a major problem, but he'll also look into some of the testing they've done to confirm it.

He did say that the drive has the intelligence to attempt to speed match to the data rate of the arriving data by slowing the motors down to prevent excessive data under-run. He thought that all generations of the drive would be able to come close to the low data rate of the Gen 2 drive, but again would confirm this with the technical team.

Whether any of this would apply to the HP drive as well I can't say, but there are significant mechanical and logic differences between the HP and "XYZ". They also use completely different head technologies and head servo-ing mechanisms.

LTO has three sets of servo tracks written down the tape when it is manufactured and as the tape moves through the drive the heads wiggle up and down as three dedicated servo heads track the pre-written servo patterns.The mechanisms that drive the head up and down are very different by manufacturer.

Jim,
Here is another article referring to HP, IBM and Quantum drives, and suggesting that system matching is critical with all of them.
http://tinyurl.com/2j9xkq
Patrick

Patrick,

Thanks. I sent this on to the folks at "XYZ" and will follow up tomorrow.

LTO 5 is planned to be at 180 MB/s native 360 MB/s compressed

It should be out around the end of the year

LTO 6 is planned to be at 270 MB/s native, 540 MB/s compressed. It's supposed to be available end of 2010

Patrick,

Talked to my technical contacts today.They say they've tested extensively and have never seen any damage whatsoever to either drive or media from shoe-shining caused by low data rate sources.

I also talked to a contact at one of the major computer consulting companies that tracks industry data. He confirmed my belief that a significant percentage of LTO drives are used to back up Windows servers.

Also, Dell sells LTO drives, and their business is largely Windows oriented.

Both IBM and HP drives have speed matching capability that lets newer generation drives run at roughly the same speed as Gen 2 drives by tracking the incoming data rate and adjusting motor speeds accordingly. Not sure about Quantum (which acquired Seagate's LTO business).

Welcome to the wild and wonderful world of storage performance claims.

I have a couple of other calls out and will update when I hear back.

OK, I've talked to quite a few folks and I have no doubt from a hardware point of view that LTO will work just fine.

Now the big question left in my mind is what software is needed to drive it and manage the archives. This could get tricky as most packages that support it tend to be fairly large/complex/expensive packages aimed at companies with an IT staff.

I'm going to do some research the next few days. I'm also going to get a small LTO library and see what has to be done to get it working well and simply on my system. Might take me a while, but since I'm such a big LTO fan, I think it's something I need to do!

Stay tuned!

Jim,
Thanks for all of year reasearch on behalf of myself and others on this list.
Patrick

Patrick,

Thanks for your kind comment. I've asked for a sample drive to experiment with and should be able to come up with one. May take a couple of weeks to find one, though. So if you don't hear anything on this for a while, don't think I've forgotten about it.