Thanks George,
I started this thread with an example of external hard drive failures. However, the RAID conversation is also related to this topic.

Seems that the best way to go around the RAID decision is to simply set the two 500GB drives to RAID0 and then use a program called "SecondCopy" to backup the newly added data everyday to a different drive. This program allows both manula one click backup as well as scheduled. This way the video editing write/read access while video editng is not compromised on that RAID0 1TB.

@ Aaron

LTO, DLT write/read access is much slower than that of optical, isn't it?

Thanks again to everyone!!!

I woudn't even think about DLT (Digital Linear Tape) anymore. It's about as dead as a technology can get and not be buried. It's been almost totally blown away by LTO (Linear Tape Open). Even Quantum (the maker of DLT) recently bought the tape piece of Seagate, which was one of the original memberts of the LTO consortium, together with IBM and HP and is now making LTO their major focus in tape.

At 120 MB per second (native non compressed) I think it is way beyound anything you'll get from an optical drive. The biggest perfomance issue with tape is the time it takes to space down the tape to where you want to start reading. (Time to first Byte)

Current LTO Generation 4 has native capacity of 800 MB per roughly 4" sq cartridge. Generation 5 will be along around the end of the year or early next at 2X the capacity and higher data rates. Generation 6 is spec'ed at 3.2 TB per cartridge (uncompressed)

The IBM/HP/Quantum consortium is committed to developing at least 6 generations of the product and nobody is thinking of stopping there, so it's a very safe bet to be around for a long time yet. Yearly sales for all vendors total over 400,000 units, another indication of why it won't disappear anytime soon.

Drives by IBM, HP, and Quantum are highly reliable. I'd avoid drives made by any other company. Dell OEMs the drives and resells them so they are easily available.

It's a super technology. There are lots of small LTO tape libraries available at reasonable prices (in the ballpark of small pro cameras) that can really help streamline any backup and archive application.

DISCLAIMER

My day job is as the business interface between two companies that have been partnered in developing the LTO family of products. I've been working on the project since the beginning.

I'm sure you meant that to be 800GB per tape, just to nitpick.

George/

George,

Right on - I meant 800GB, I'm sitting in a hotel room in Tokyo and am fumble fingering the in room PC keyboard

Jim,
Excelent info! Thanks so much.
So, how long would it take to write, say, 60GB of data to an LTO gen 4 tape? Does it matter what brand gen 4 drive you have? If choice is among the big guys you mentioned - not OEM's of course.

Also, what do you mean by performance issue in regards to tape vs. optical media. Isn't it that you just insert the tape and then go to windows explorer and start browsing files right away?

Thanks again!

Tape media, including LTO, is cheap (at least per GB) but a bit cumbersome to handle/operate.

I would suggest to look at professional optical disc storage, like the SOny Professional Disc. It is not as fast as tape in write spead, but it stores 23 or 50Gb of data with a shelf life rated at >50 years. And you can just browse it like you do a CD/DVD.

They make a good project archive medium.

LTO has several advantages, but I believe it is less suitable for archiving unless you have enormous projects and a good climate controlled storage facility. Tape is more fragile than optical media especially when it comes to heat and moisture.

George/

Yeah, I checked the prices and was astounded that the cheapest LTO4 drive goes for like $3200. The media is cheap but the drives are ridicuously expensive and huge too. I think Blue-Ray has promise value-wise.

The most impotant thing is to have some kind of formal archive/back up plan and to follow it.

After that, it's all a matter of degree.

I don't know that I'd consider LTO for all applications.

On the other hand, if it's appropriate for your situation, it's hard to beat.

Maybe a rough rule of thumb would be that if the drive is less expensive than your camera or your sound setup, then you should look at it pretty seriously, otherwise optical might be OK. Another criteria might be the volume of stuff that you want to archive, Considering that one LTO cartridge can hold 50 or 60 MiniDV cartridges and the number will go into the hundreds in the next couple of years, I think people should look at it for any kind of high volume usage. Data rate is also a big factor if you're archiving whole projects.

If you're using HD and something like high or film scan level CineForm intermediates - or even uncompressed, then I'd really get serious about considering LTO

Re fragility - not sure I'd consider the media fragile. We re-use cartridges 5,000 times apiece in our drive test line and we've seen some that will be fine after 20,000 load - unload cycles. Normal home environments are climate controlled enough from everything I've seen. (Well, maybe a non air conditioned home in Singapore or Tucson might be stretching it a bit!) We also test the drives at a lot of load unload cycles each before OK-ing them for shipment. Some of the networks in Japan are using LTO as their archive of everything they broadcast.

Cleanilness is impportant - you don't want to get dirt inside the cartridge. On the other hand, there's a spring loaded door that keeps the cartridge closed so that no tape is exposed.

In all, it's quite a different animal from MiniDV tape. Which is all I really wanted to say, as "tape" gets a bad reputation based on characteristics of MiniDV tape and it's important to realize that there's quite a spectrum of "tape"

I built an extra computer that acts strictly as storage. I bought the cheapest Intel CPU and decent motherboard. Rocketraid makes some nice cards that can handle 2-8 drives. I then built a RAID 5 out of 8 - 500 GB drives. Thats just over 3.2 TB of storage. This machine is turned on/off only when I know that I need something off of it. I did have 1 drive die recently, but the computer still booted up, told me of the drive error and it was replaced. I was even able to access the data while the new drive was being re-built.

I used to rely on RAID but one day the RAID controller card in the PC died and trashed both disks. In a real commercial array they will usually have provision for dual controllers.

Even with RAID, there should still be a separate back up and archive process.

Any advice about implementing LTO with a desktop computer? HP have told me they don't support LTO drives with desktops. I have a desktop (core 2 duo) running either XP or Vista. My hard drives are SATA or eSATA. I think LTO-3 may actually outperform my disk drives, and potentially damage the tape drive or tape, as it requires a MINIMUM data transfer of 27Mb/second. LTO-2 might be appropriate for data rate, and has a maximum data storage of 200Gb uncompressed per cartridge; presumably uncompressed is what I would want for video files. The drives are all either SCSI or SAS interface, so I assume I would need a PCI express card to give me the correct interface. I was thinking of the Adaptec 2250300, which claims: "PCIe x1 host connectivity, Compatibility with SCSI and tape hard drives, Ultra320 SCSI single channel.
Total cost in UK about £1,000 for hardware, and tapes cost £20 for 200Gb. Has anyone actual implemented something like this. I don't want to buy the hardware, and then find it doesn't work.
Patrick

Now you're embarrassing me!

I've been working on LTO development (but primarily on the business side) for 10 years or so, but can't keep the various models straight in my head.

As well as "little details" like the best way to actually hook one up to a PC, what application to use for it, etc. All these minor details that are so necessary to actually USE one.

I know a lot about the structure of the roller bearings, what kind of coating we apply to avoid having the metal roller flanges cut by the abrasive edge of the tape, what kind of motors we use where and why, etc, I'm just a bit DULL when it comes to plugging one in and using it!

I have a call in to the head of the LTO drive engineering team at the XYZ Corporation (3 letters, so it isn't HP!) and I'm going to try to get with him in the next day or so and try to address your questions and comments.

Patrick,

One additional comment:

Re compression, the LTO compression algorithm is completely lossless, so it won't mess up your data in any way. I doubt you'll get much of any compression of video, but I"m also pretty confident that LTO compression won't make the files bigger (it can happen in some pathological cases with some algorithms)

Normal coded info (ie text, spreadsheets etc) usually will compress by a factor of 2 to 3 on LTO but everybody just claims 2X.

Sort of in the category of more than you probably want to know, LTO drives have a set of write heads and a separate set of read heads. During write, the data as written to tape is automatically read back and compared. if there is any discrepancy, the write is simply terminated, the drive skips forward a predetermined distance, and does a re-write of the block in question. if the write completes normally, various block end and error correcting data is appended to the block.

On readback, if a block terminates without the appropriate end of block info, the drive knows that the block is bad and throws it away and continues reading down the tape until it picks up the correctly written copy of the block.

All of this makes for a busy set of logic chips - there will be up to a total of 32 (and soon many more) reads/writes going at once. We're hard at work on Gen 5 which will have a native capacity of 1.2 TB per cartridge.

Because of the writes going in parallel, tape can be written at data rates in excess of what a single disk drive can sustain. This is normal and the engineers allowed for it in the design, so I wouldn't worry about damage to the drive.

It's a completely different animal than the helical scan drives used in video applications.

Now if I just knew a few details about how to actually use one...

Hope to have answers for your questions tomorrow or the next day.

Jim,
Thanks for looking into this.

This is what HP says about data throughput, which is what made me nervous:
"The tape drive should be operating at or above its lowest streaming data rate to achieve the best life for the head, mechanism, and tape media. If the data is not sent fast enough, the internal buffer will empty and the drive will not write a continuous stream of data. At that point, the drive will start exhibiting what is called head repositioning. Head repositioning is also known as "shoe shining" and it causes excessive wear on the tape media, on the tape drive heads, and on the mechanical tape drive components."

For LTO-2 Lowest native data rate is 10Mb/s, LTO-3 26Mb/s, LTO-4 40Mb/s.

http://tinyurl.com/593ys2

LTO-2 at 10Mb/s is presumably not a problem with a single SATA drive, but I am not sure it is possible to guarantee 26Mb/s for LTO-3 at all times. LTO-2 tapes store 200Gb uncompressed and cost the same as the LTO-3 tapes which store 400Gb.

Thanks for your help,

Patrick

It's also called back-hitching. The term "shoe shining" comes from the behavior of the drive when incoming data doesn't keep up. When the drive runs out of data it stops, but stopping isn't instantaneous, so it will back up and then read forward slowly to find the end of the previously written data before it starts writing again. The back and forth motion makes people think of the old fashioned way people shined their shoes by rubbing a cloth back and forth across the shoe surface.