hello all,
I have a question about archiving our masters and MXF files from all the projects we shots using the P2 workflow.
I'm convinced DLT is the way to go, instead of buying more hard drive that take space and are more incline to failure.
Our IT guy, is convinced that hard drive are better because it's a cheaper solution.
he also think that we should compress the data ( what I mean by compressing is using ZIP or RAR or something like that)
Now, I've always heard this is a big no no in a post environment.
heard of stories where the data ended up corrupted when de-compressed.

My understanding of these compression format is very limited, this why I turn to all of you,
Am I crazy to think this is not a good idea?
loved to hear some real life experiences.

DLT is great. Just copy to it and use it when you need. Hard drives are okayish,but be sure to have two copies in two different HDD's. use them in regular intervals (once every month, do a virus scan on them or something that makes the drive mechanism spin), in order to prevent the drive motor to die. Store both mediums (DLT or HDD's) in a cool, dry and safe place (Humidity boxes).

With DLT, you'll feel a lot more safe than with HDD's, but its a lot more expensive than the HDD's. DLT is the industry standard, and used by the big corporations for their files backups.

Have you considered getting an Iomega REV drives? Those sound right. the company I work for has just invested on them. Haven't arrived yet, but sounds good on paper.

And don't forget Blu-ray. It should be a nice alternative for storage too.

Don't compress anything. Leave everything has they are. When you compress, you are changing something. If anything goes wrong in the compression and decompression process, its game over for your footage.

DLT is great. Just copy to it and use it when you need. Hard drives are okayish,but be sure to have two copies in two different HDD's. use them in regular intervals (once every month, do a virus scan on them or something that makes the drive mechanism spin), in order to prevent the drive motor to die. Store both mediums (DLT or HDD's) in a cool, dry and safe place (Humidity boxes).

With DLT, you'll feel a lot more safe than with HDD's, but its a lot more expensive than the HDD's. DLT is the industry standard, and used by the big corporations for their files backups.

Have you considered getting an Iomega REV drives? Those sound right. the company I work for has just invested on them. Haven't arrived yet, but sounds good on paper.


Thanks sergio
That confirms what I thought about compressing the data.
I did take a look at Rev pro media, sounds interesting but unfortunately the media can only hold 35gig max (as of today) and cost around $60 per disk.
DLT tape is 300gig and cost around $90.
I just did an interesting calculation to compare prices between hard drive and DLT backup.
Based on $1000 per TB which is what hard drive cost now.
and based on the price of the new Quantum drive $7000 + $90 per DLT tape
here's what I came up to.
Capacity(TB)---HD costs------DLT Cost
---1------------ $1,000----------$7,270
---2-------------$2,000----------$7,540
---3-------------$3,000----------$7,810
---4-------------$4,000----------$8,080
---5-------------$5,000----------$8,350
---6-------------$6,000----------$8,620
---7-------------$7,000----------$8,890
---8-------------$8,000----------$9,160
---9-------------$9,000----------$9,430
---10-----------$10,000----------$9,700
---11-----------$11,000----------$9,970
---13-----------$13,000---------$10,510
---14-----------$14,000---------$10,780
---15-----------$15,000---------$11,050
---16-----------$16,000---------$11,320
---17-----------$17,000---------$11,590
---18-----------$18,000---------$11,860
---19-----------$19,000---------$12,130
---20-----------$20,000---------$12,400
past 10TB DLT becomes much cheaper, for 20TB you pay $20k in hard drive vs $12,400 for DLT.
+ I'm not counting the time and energy (AC Power) it takes to run those hard drive.
We have at our facility around 7TB that needs to be backed up cause our Raids are getting full, I think DLT is definitely the winner.
thanks for your feddback

I transfer my P2 footage to a mirrored RAID. It's based on Firmtek hardware. The drives are IBM/Hitachi SATAs on sleds which allow them to be hot-swappable. Ultimate capacity is unlimited except for one's budget.

In the event a drive dies, the second one continues to operate undisturbed. I actually did a test where I pulled a drive to simulate a drive failure while data was being written and the "good" drive continued to operate normally.

Plugging in the "dead" drive and reformatting it, then mirroring it, restored things back to normal in this test.

Writing data to a mirrored RAID is, essentially, making an instant backup. Storage cost is about $1.15 per minute or so. Including the cost of the drives, sleds and shipping.

Fast, reliable and economical.

Those hard drive prices are a little high. A 300GB Maxtor DiamondMax w/16MB buffer goes for $119. That makes it aprox $440 per usable TB. So 20Tb is only $8,800.

Those price are based on Raid NOT just the hard drive.
Apple Raid are $12,999 for 7TB
LaCie Raid $1299 for 1 TB
Granite Digital $5955 for 6 TB

If you want secure backup Raid are the only way.
so no, those prices are correct.
e.

RAID is inherently insecure and must itself have redundancy either built into the RAID protocol (ie RAID 50) or with a separate RAID system (ie. RAID 1). It seems illogical to use RAID for secure backup of master files.

It seems to me that the logical use of a hard drive for backup is to plug in a new unit, fill it with data (from a RAID), disconnect it, label it and put it in a secure storage space, the same as you would a master tape. Should that one brand new drive fail the data is much more likely to be recoverable than if it is spread across a RAID system.

This is just my opinion of corse Eric. I've been following and respecting your advise on this forum for quite a while and I offer this opinion in the spirit of contributing, not confrontation.

In the event a drive dies, the second one continues to operate undisturbed.
I really don't see this . . .

The whole idea of a raid is for speed - that's why they write to both disks at once. So how can this be viable - I'd love to know because raids don't write one file to one disk and another to the other disk - they spread the data between both at the same time - as I said, for speed.

And I agree with Martin - raids are for Editing rather than storage.

If I'm wrong, please educate me!

Andy

You're wrong! -;)

There are many levels and functions for RAID. What you describe is RAID 0, Then speed is the consideration. Than you have a total redundant RAID where data is written to two (sets of RAID 0 - sometimes) disks to have total backup. Then you sorta have everything inbetween. Some optimized for speed, others for security, some for both (Like RAID 5)

Gunleik

RAID 0 for speed.
RAID 1 for reliability.

The RAID I'm using for storage of master files is a level 1 RAID.

It writes identical data to two disks simultaneously.

This type of RAID is used for systems where reliability is a must. Such as banking data, air traffic control, etc. It's been around for quite sometime actually.

As mentioned before, I did a test where I pulled one of the drives while data was being written (to simulate a failed drive) and the data remained intact on the drive that was still running.

After the pulled drive was reformatted (to simulate installing a new drive), the data was re-mirrored and the system was back to normal.

With a RAID 1 you don't have to waste time writing to one set of drives, then to a second set because you're always writing to two drives simultaneously.

RAID 0 for speed.
RAID 1 for reliability. . .

. . . With a RAID 1 you don't have to waste time writing to one set of drives, then to a second set because you're always writing to two drives simultaneously.
Wow!
You learn something every day!

Thanks for that. I'll keep it in mind next time I need a new External HD.

Andy

RAID is inherently insecure and must itself have redundancy either built into the RAID protocol (ie RAID 50) or with a separate RAID system (ie. RAID 0). It seems illogical to use RAID for secure backup of master files.

It seems to me that the logical use of a hard drive for backup is to plug in a new unit, fill it with data (from a RAID), disconnect it, label it and put it in a secure storage space, the same as you would a master tape. Should that one brand new drive fail the data is much more likely to be recoverable than if it is spread across a RAID system.

This is just my opinion of corse Eric. I've been following and respecting your advise on this forum for quite a while and I offer this opinion in the spirit of contributing, not confrontation.
hey, no worries, this is exactly what this forum is about, I think something, you think different, sometimes I'm wrong, sometimes you are, in the end we both learn something.
e.

You're right Eric and this time I have to correct myself. I edited my posting from so that it correctly states RAID 1 instead of RAID 0 for having built in redundancy. I knew that, just typed it wrong.

For those following along generally a RAID 1 will slow down your system as opposed to RAID 0 which will increase processing speed. RAID 1 is not recommended for video editing where processing speed is important. I run RAID 0 with 3 300GB hard drives and use a program called Carbon Clone Copy to copy the data on the RAID onto a single 750GB hard drive. The RAID gives me speed and the single hard drive gives me reliable back up.

Here's the cost of a Firmtek solution:

Firmtek SeriTek/2eVEN4 External Bundle (External 4-Bay Enclosure & 4-Port PCI Host Adapter): $579

4 500-GB Hitachi/IBM SATA drives: $185 each x 4 = $740

Total startup cost for 2 terabytes of storage: $1,319.

Each additional terabyte requires a pair of Hitachi SATA drives and two Seritek "trays".
Drives: $185 x 2 = $370
Trays: $23 x 2 = $46.
Total : $416.

To get a total of 20 terabytes:
First two TB = $1,319
18 additional TB = $7,488
Total = $8,807

These drives can be configured in a variety of ways. Single drives, striped pair (RAID 0) or mirrored pair (RAID 1). And they can be set up in combinations as well. You can have one pair set up as RAID 0 while the other pair is RAID 1 in the same enclosure. Or you can go nuts and stripe all four drives as a single high-speed RAID.

Keep in mind that this cost analysis is for 20 TB of straight storage, not duplicated RAID 1 (mirrored drives). If you mirror the drives, the cost per usable terabyte doubles.

What a hard-drive system gives you, that DLTs won't, is random access. Locate that clip with a catalog system, plug in the drive pair, and grab it. No need to fast forward or shuttle a tape.

With any archive system, what's vitally important are backups. Even tapes fail. So those things have to be taken into consideration when designing a system for the long term.

And then there's upgrades. When something else takes the place of SATA or any other storage medium, how long will it take to migrate that data to a newer medium? For that matter, how many of us have migrated older tape formats to newer ones?

It's all scary stuff when you sit and really think about it.

Really scary, Eric. Bottom line, what we all really needed is a DVCPRO HD deck, but that's just too expensive. How about renting a deck, buying some tapes and copy your cards back to p2, then use the camera's FW port to copy to the deck? I really don't know how much a dvcpro HD tape goes for, but, concerning safety, having the footage on a playable tape were you can visually see the footage on a deck, always seemed the most reliable...

My ideal solution was to have a MXF backup and and a dvcpro HD TAPE backup- but that's insane and overbudget...

Really scary, Eric. Bottom line, what we all really needed is a DVCPRO HD deck, but that's just too expensive. How about renting a deck, buying some tapes and copy your cards back to p2, then use the camera's FW port to copy to the deck? I really don't know how much a dvcpro HD tape goes for, but, concerning safety, having the footage on a playable tape were you can visually see the footage on a deck, always seemed the most reliable...

My ideal solution was to have a MXF backup and and a dvcpro HD TAPE backup- but that's insane and overbudget...
Yep, as much as I like the P2 workflow, archival is definitelly the weak link.
I decided to invest in the new Quantum SDLT 600a (http://www.quantum.com/Products/TapeDrives/DLT/SDLT600A/Index.aspx)
It's not ideal but it is to me the safest solution for long term (30 years) backup.
Make a tape and store on a shelf in a dry place, it will also save space in our server room from all the Raid we would have needed over time.
So, in the end it's ironical to me to still end in a tape in this tape-less workflow.
Again, this is what I think will work best for our post house, other people might prefer backing up to Raid or single hardrive, after all, it's a free country :)
e.

A very interesting thread.

And here's a very important point to remember: RAID 1 DOES NOT EQUAL BACKUP. RAID 1 is good to make it less likely that a drive failure will cause you to lose data, however, if you delete a file from a RAID 1 volume by mistake, it's gone.

A truly reliable ARCHIVE requires the use of something that lives offline that is not prone to accidental deletion or electro-mechanical device failure. Like DLT and LTO tape, or DVD-R, or Blu-Ray are all reasonably good archive mediums. It's also a good idea to keep them off site.

For ultimate protection, follow the rule of three: (1) working copy on hard drive, (2) hot backup on second hard drive (ideally set volume to READ-ONLY), (3) archive on tape. Statistically, it would be a very rare event for all three copies to become unreadable at the same time.

And...
One way to make RAID 1 ultra reliable is to use three disks and rotate one in and out of an offsite backup location. This third drive would not be prone to accidental human error deletions.

Yep, as much as I like the P2 workflow, archival is definitelly the weak link.
I decided to invest in the new Quantum SDLT 600a (http://www.quantum.com/Products/TapeDrives/DLT/SDLT600A/Index.aspx)
It's not ideal but it is to me the safest solution for long term (30 years) backup.
Make a tape and store on a shelf in a dry place, it will also save space in our server room from all the Raid we would have needed over time.
So, in the end it's ironical to me to still end in a tape in this tape-less workflow.
Again, this is what I think will work best for our post house, other people might prefer backing up to Raid or single hardrive, after all, it's a free country :)
e.

Hello Eric,
How much do you shoot? If P2 is a problem to archive how do you think the RED users will fare when they need to back-up 160 gig drives vs. a 4 gig P2 card?

We shoot quite a lot, long interviews, multicam concerts, and some narrative.
Just to give you an idea, I worked on a project last week where we ended up with 146 8gig content folders between 3 cameras. that's 1TB of masters right there.
Obviously RED users will face the same dilemma X 10.
The one thing I think most indie filmmaker (myself included) don't realize, is that a reliable archival process of a tapeless workflow is quite expensive in the end.
Unless you're David Fincher and have big studio money to back you off.
here's a interesting article on his latest film and tapeless workflow.http://digitalcontentproducer.com/videoedsys/revfeat/video_going_tapeless/

A very interesting thread.

And here's a very important point to remember: RAID 1 DOES NOT EQUAL BACKUP. RAID 1 is good to make it less likely that a drive failure will cause you to lose data, however, if you delete a file from a RAID 1 volume by mistake, it's gone.

A truly reliable ARCHIVE requires the use of something that lives offline that is not prone to accidental deletion or electro-mechanical device failure. Like DLT and LTO tape, or DVD-R, or Blu-Ray are all reasonably good archive mediums. It's also a good idea to keep them off site.

For ultimate protection, follow the rule of three: (1) working copy on hard drive, (2) hot backup on second hard drive (ideally set volume to READ-ONLY), (3) archive on tape. Statistically, it would be a very rare event for all three copies to become unreadable at the same time.

And...
One way to make RAID 1 ultra reliable is to use three disks and rotate one in and out of an offsite backup location. This third drive would not be prone to accidental human error deletions.
David,
I just checked out Kino-Eye.com for the first time, great blog! full a great articles. thanks for the work.

We shoot quite a lot, long interviews, multicam concerts, and some narrative.
Just to give you an idea, I worked on a project last week where we ended up with 146 8gig content folders between 3 cameras. that's 1TB of masters right there.
Obviously RED users will face the same dilemma X 10.
The one thing I think most indie filmmaker (myself included) don't realize, is that a reliable archival process of a tapeless workflow is quite expensive in the end.
Unless you're David Fincher and have big studio money to back you off.
here's a interesting article on his latest film and tapeless workflow.http://digitalcontentproducer.com/videoedsys/revfeat/video_going_tapeless/

Yes I read that.....your heading into the great unknown....
Hopefully storage will only go down in price as capacity goes up.

For the indie producer, tape is still a viable "archive" format, however, with the constant drop in hard drive prices and Blu-Ray on the verge of becoming standard read/write media on new computers, I will say that 2007 will be "the year of tapeless workflow for indies" and I already think 2006 was "the year of tapeless workflow" for all but indie and low-budget producers.

Some hold on to tape, however, I live most comfortable on the leading edge just behind the bleeding edge. You sometimes get spashed with the blood but it's most often not your own. But I've seen old tapes go through video decks leaving much of the oxide inside the deck. No love for videotape here. On the other hand, hard drives die.

Maybe we can get 10 good years from Blu-Ray and then copy to something else, if what we have is worth preserving. Some say properly stored tapes are good for decades, but who's going to have the drives around? It's all a nightmare. Just ask any media archivist.

David,
I just checked out Kino-Eye.com for the first time, great blog! full a great articles. thanks for the work. Thanks for the shout out :-)

What about the so-called "100 year" achival gold DVD-R's?

Anyone have info on how "archival" these really are?

I've had a commercially pressed CD go bad on me, and that CD wasn't handled all that much. I've also had various DVD-R's become unreliable over a few years.

Hard drives have failed. Tapes have gone bad.

The best bet is distributed storage. It would be impractical to make multiple copies of everything and put everything in dispersed locations, but for the really important stuff you might want to make copies onto a couple of different media just in case. And have a copy located elsewhere should a small disaster occur in your neighborhood.

What about the so-called "100 year" achival gold DVD-R's? Anyone have info on how "archival" these really are?
Ask me in a hundred years.

Seriously, in theory the use of gold instead of silver or aluminum as the reflective layer in a DVD provides better resistance to chemical breakdown from oxidation, which is a significant cause of disc failure, however, disc longevity is also threatened by delamination of the layers and fading of the dyes.

By using a combination of gold for the reflective layer, high quality bonding agents, and fade resistant dyes (e.g. Phthalocyanine) for the coloring layer, an "archive grade" DVD-R can be made that will probably last over 100 years. MAM-A (formerly known as Mitsui) claims to to make such a thing in their (relatively) new 4.7GB 8x Gold "Archive Grade" DVD-R. Controlling exposure to humidity, temperature, and light are still important factors in increasing disc life.

Since accelerated testing in the lab is never the same as the ravages of time, we'll only know for sure a hundred years from now, but by then, I'm confident we'll have a reliable alternative medium to which to copy our data to, assuming global warming or thermonuclear war does not make the whole practice of archiving moot.

You might be interested to know that the Library of Congress and National Institute of Standards and Technology (NIST) have been involved in an Optical Media Longevity Study (http://www.itl.nist.gov/div895/loc/overview.html) and public test results will be available when the study is completed.

I've had a commercially pressed CD go bad on me, and that CD wasn't handled all that much. I've also had various DVD-R's become unreliable over a few years.

Hard drives have failed. Tapes have gone bad.

The best bet is distributed storage. It would be impractical to make multiple copies of everything and put everything in dispersed locations, but for the really important stuff you might want to make copies onto a couple of different media just in case. And have a copy located elsewhere should a small disaster occur in your neighborhood.


I see you are from Honolulu, I lived in Hilo on the Big Island for a while. The salt air and tropic humidity wreck havoc on anything there!

Of course "archival" grade anything can easily be ruined, when I took my still photography courses in college we used archival grade everything, but if you didn't rinse your prints long enough a year later, your prints turning brown. (I ALWAYS rinsed long enough, others...not so much)

I guess my question is more: has anyone used the "archival" gold DVD-r's or had any experiance with them?

sorry for the confusion.
Phil

[...] I guess my question is more: has anyone used the "archival" gold DVD-r's or had any experiance with them? [...] I've burned some gold discs, however, I will probably not know in my lifetime whether they are truly archival or not. That's for the next generation to decide/discover.

FWIW, I have some "archival 100+ year rated discs" that I've burned as far back as '93. They all still work... Then again, All my standard dye-based discs from that era still work too... With the exception of one 50-pack of discs from about '95 that didn't last. But those were weird -- half of them didn't work to start with.


For now, nobody knows anything and all these lifespans claimed by manufacturers are just a guess.

[...] for now, nobody knows anything and all these lifespans claimed by manufacturers are just a guess. It's true that lifetime estimates are best-guesses based on accelerated longevity testing, however, I would not go so far as to say that nobody knows anything.

We know that some dyes are more fade resistant than other. We know that some bonding techniques are better than other. We know that gold makes a better reflective surface for archival storage. We know that not all disks are created equal and that we should choose the discs we use based on how they are manufactured and that MAM-A (formerly known as Mitsui) makes some very high-quality archive-grade discs.

And check out the link to the Optical Media Longevity Study in my earlier post. They will be publishing their results in the near future.