Storage- How Are You Handling 5D files
 

Okay, in the HVX200 intro days, this was a constant discussion. With the 5D and 7D, we are facing that again. Having filled a couple of Terabyte disks, I am trying to figure how to go with storage next. I would like to free up those disks, and wonder what others are doing to archive footage. Though I convert to Cineform, I will be saving only the original files. I can always reencode with Cineform, if needed.

Any one have any mass storage ideas at this point ? Is Blue Ray data a possibility ? Stable ?

My approach is to convert H.264 files with MPEG Streamclip to Prores 422 then store those on a double sided dvd.I then assign a volume name and number to the dvd and log basic info into a database. This gives me 8 gigs of storage. For Larger projects Blu-ray would be a better way to do it.

Right now I'm just keeping my source & other project related files on a mirrored array and all the prores files on a raid0. Closing in on about 4TB total from this past year and I just figure I'll add another similar pair of drives to carry me through the next year. At a cost of just a few hundred dollars it seems the cheapest & easiest way to go.

I just buy 1 Terabyte external eSata drives like they are going out of style, LOL. It's great. I just dropped an eSata card in my PCI slot and now these puppies hum pretty fast once connected externally. And these days, they only cost like $129 or somesuch. Anyway, I keep two copies of the originals (like you I discard the intermediates and keep the untouched originals), with the second kept off site.

This is what I'm doing. Seems like the best approach. I've never quite trusted DVDs for long term storage, and while I love Blu-ray video I'm not yet confident in that media for data storage.

cheap internal drives -> external drives?
 

I'm running into the same problem with storage, especially when you have to change to PreRes for editing.

I've seen large, cheap internal drives for sale (1.5TB+ at $129). Is it possible to put one of those into what's called an "enclosure" and use that? If the combo had e-Sata, it would be a lot of space for not too much money.

Optical BR and HDDs
 

I back up "critical" footage to DVD, DVD DL, and BluRay disks. Bought an internal BR drive from MCE tech and love it to pieces (plugged right in no issues at all, no additional sw to load)! I find BR blanks for as low as $3 25gb single on the bay. I heard long term is not a good idea on HDD.

At $3 25GB for long term storage is a great solution. Burn MUCH faster than I thought it would. The simplicity is awesome with the latest tiger updates allowing you to create a BR burn folder right on the desktop! Drag and drop 48GB into a burn folder and make some coffee! or keep working because it will let you do that too :)

Gotta love those deals on HDD though. I only store the native files like you mentioned because you can always transcode again.

I bought 50GB ink jet printable BR disks for under $6 delivered..... brand ..starts with an S and ends with an ony BR storage is a solution now that the media is much cheaper than it was only a few months ago.

Well for the next Terabyte, I bought a Vantec NexStar hard drive dock, for $40, and then bought an Hitachi terabyte drive for $70, both at Frys. The Dock works like a popping the hard drive in like a toaster. It all hooks into the SATA set up on my mother board. Should be just like an internal drive, except you can change drive easily by just ejecting one and replacing with another. I will be installing this weekend.

See picture below.

Currently I use three hard-drives, one work-drive and two containing redundant backups -- my internal drive is my work-drive, and then I have two external ones for backup -- one of the backup-drives I always keep in my car, and the other I have locked in at work. Over the years I've lost quite a bit of data... and each time I've said "never again".

-- peer

Long term storage (if that what you need) on an HDD is looking for trouble. Between reports that magnetic info disipates over time (unless the disk is powered up and files structures are verified) and the always imminent mechanical failure drive are renowned for. If you work in a video production facility like I do, those are not an option. I like Mikes approach, but believe it or not the manner that is accepeted to be most reliable is very expensive file backup to data tape, which sounds to damn slow to me.

I can remember when the HVX200 was the rage, and all were trying to figure out what to do. I finally suggested tape backup..

archival blue ray discs
 

Delkin Devices makes a gold blue ray disc which they "guarantee" to store your data for 200 years. The prices listed on the link below are a around a dollar a Gig which is significantly more expensive than some of the other solutions mentioned but if they do hold data for 200 years then they might even be cheaper in the long term.

I would love to hear anyone's thoughts on this product.

Blu-ray Archival Gold Recordable Media, The 200 Year Disc – Delkin Devices

This might be the most reliable option
 

Have a look at the Sony XD Deck PDWU1
Sony Product Detail Page - PDWU1

I have been working with XDHD422 a lot and I have to say, it's a solid piece of kit and codec.

The discs are more or less a similar price to Bluray discs

The only thing is that you'll have to convert everything to XD files but for me thats a positive not a negitive

Hope this helps

G

I would stick to the blu-ray approach. That way if you have Prores clips or movies (or any other sources from clients other then XDCAM) no conversion is needed for archiving.

I'm using hard drives through SATA external storage bays and docks. Online drives have 1 copy of current projects. I keep 2 separate backups of those projects - one onsite, one offsite. Then archived projects I put on drives and keep onsite, and have 1 backup of everything offsite. I'll also make DVDs of final exports and compressed versions. It also helps to have things published and distributed on disc or online and that's another way that at least distributed copies exist.

If all that fails? Well screw it, I tried. Life is too short to worry endlessly over this stuff. Find a reasonably convenient method you'll actually use and follow, and move on.

-steev

Yeah, it's important to have offsite backups. Hard drives and optical media (DVDs and CDs) are both imperfect; there's no great archival solution for digital files. There's a good argument for having your files on two different types of media, as they each have their own strengths and weaknesses. So my advice is one copy of each file on a hard drive, and another on a DVD or Blu-ray disc. And be sure to store one set offsite.

As for hard drives, I agree with the others who have suggested the hard drive docks, so you can buy multiple internal drives without buying multiple enclosures. Regarding DVD-Rs, Taiyo-Yuden media is highly rated, and not super expensive. Just be sure to verify your burns, don't burn at max. speed, and don't write on the disc (you can use an archival pen to write the disc number on the clear part around the disc's hole).

There might be a good argument for capturing to mini-dv tape as well (from computer).

Why mirrored?

it duplicates the same data from the original source. Two mirrors are better than one right? You break one, you can use the other to put your makeup on right? j/k

thanks.

I keep two copies of my source video on my main machine, and my backup file server. My file server is a Norco DS-4220 case, which is common with the HD movie enthusiasts on sites like AVSforum.com and Hardforums.com. Some are actually video editors, but many are movie guys with 40TB+ of space. It's a 4U rack mountable case, and fits 20 SATA hard drives via a SAS backplane. I put a Gigabyte AMD 785G motherboard and a low power Athlon 64 X2 5050e cpu.

The gigabyte motherboard has at least 6 SATA ports, and at least two PCIE x16 slots. So you canput in cheap Supermicro AOC-SASLP-MV8 cards, which takes two SFF8087 connectors, or basically 8 drives. The case has essentially 5 backplanes, so you get two of the Supermicro SASLP-MV8 cards, which takes care of 4 backplanes, or 16 drives. Then you get a SATA to SAS reverse breakout cable, which will plug into the 5th backplane, and your sata ports on the motherboard, adding to the total of 20 SATA drives.

Some good reviews of the case are here. You can get the case through newegg.com, or mwave.com through its ebay store for around $300. The AOC-SASLP-MV8 cards are around $110 through provantage.com (and also get SAS-SAS SFF8087 cables and SATA-SAS reverse breakout cables there too).

I have four 1.5TB drives, and three 2TB drives in there, with 13 SATA hard drive trays left for tons of expansion. What's interesting is that My HDV and AVCHD files (old Sony HDR-FX7 and Panasonic AG-HMC150, both around 21Mbits/sec in 1440x1080i or 1920x1080 PH mode) are a little smaller than my old 25Mbit DV files from a DVX100 and DVC30 (typically 12GB/hour). The 5D Mark II files are around 40Mbits/sec, or 5MB/sec (18GB/hour approximately), still much smaller than say a 100Mbit P2 DVCPRO HD files. A single 2TB drive can fit somewhere in the neighborhood of 100 hours of raw 5D Mark II h.264 MOV files, 160 hours of raw DV files, and between 180-200 hours of raw HDV and raw 21MBit/AVCHD files (rough rough estimates, don't quote me).

More links for the case.

Hands On: Norco RPC-4220 Rackmount Chassis | We Got Served - Windows Home Server & Your Digital Home

New Norco RPC-4220 Server Case pictures - AVS Forum

Data Storage Systems - [H]ard|Forum

i use a similar device... as for storage i still have a few 700giggers lying around... makes backup easy. I copy the files to 2 HD drives tho, so that i still have the data if one of the HDs fail.

As for Allans fileserver: i have a very similar setup, with a Chenbro 3U (16xS-ATA) on a fancy hardware RAID6 controller. Bad part about that experience was that my RAID controller decided to die and take my array with it. So if you go for big fileservers, my recommendation is that you do make copies somewhere else (just like Allan does)... RAID servers are only for uptime, never for data security :)

A expensive $500 hardware raid card is nice because you have redundancy and can create very large partitions (10TB+). The complexity of setup and configuration can be difficult. The cost of the hardware raid cards (decent ones with dual core Intel IOP cards are well over $500 like Areca and Adaptec) plus the loss of capacity due to parity is pretty high.

The Supermicro SAS cards are around $100, and you need only two or three of them in this 4U Norco case. They are cheap enough, and with the money saved over true Hardware RAID, some people have essentially two Norco cases, one backing up the other one (40 disks max, but in effect almost RAID 1 without the synchronization realtime).

Because I don't trust a single drive for mid-term (next few years) storage of source files, and don't want to take the time to burn everything to optical. Raid1 is simple & cheap and doesn't require any additional work once it's set up. I'm sure optical is a better choice for long term storage, but I'm also sure that our capacity & format choices will be significantly different in 5 years then they are now so - that's the archival window I'm most concerned with at this point.

I think it's a great idea to not rely on a single hard drive for storage, but RAID only without offsite storage is risky. If the RAID device gets stolen or damaged from a flood/fire/power surge or corruption, you've lost your files. To me it makes more sense to have one copy in the office, and another offsite, preferably on two types of media (optical/hard drive/tape).

That's why I try to always keep an updated redundant hard drive backup in my car -- don't think it's very likely my car gets stolen the very same time my office burns down.

-- peer

Corporate policy where I work forbids archive back up to optical discs. CDs and DVDs have proven to be far too unreliable.

I have experienced many corrupted 5 to 10 year old discs - and one was faulty the very next day !!! - even though it tested ok the previous night !!! (was a cheap brand). A computer guru mate ran a data recovery app that continuosly attempted to read the lost files for 48 hours, and approx. 60% of the data was recovered.

Do I trust optical media ? - NO. The dyes degrade over time.

I wholeheartedly agree. Optical has been bad to me, even when using quality dvd-r's.

As stated above, HD's don't strike me as safe for long-term storage either. The problem are the disk head readers, if they dont start up correctly, the HD can't be read anymore.

This whole situation will change when SSD is getting cheaper. Heck, i have to admit, i'm already contemplating getting 64gbyte USB sticks, or some of the cheaper SSD "harddrives" for really reliable long-term backups... as flash-based storage degrades only on writes, not on reads.

But for now, price-wise, tape is still interesting for long-term storage... LTO-2 or LTO-3, depending on your data amounts. But even those need recopying to new tapes every 5 years or so. Keeping 2 copies can be handy... and the tape drives can be finicky.

Pick your poison :)

I wouldn't necessarily hold up SSD as the answer. Check out this recent SlashDot story:
Slashdot Ask Slashdot Story | Reliability of PC Flash SSDs?


Beyond that, my motto is more backups, at least two if not three, and then to migrate forward with technology as it happens. It always seems that right when I'm about to burst my seems, the latest large drive comes out, LOL. I'm kind of praying for some insane 10T drives to drop in the next few years where I'll just migrate off my old three copies onto new ones over time.

Biggest thing in my book is don't rely on any one copy, and make sure something is off site, and check your backups on occasion.

BTW, I personally agree that tape is awesome, especially when verified, but it's too expensive and too much of a PITA for the solo hobbiest. If I made my living on this, I would seriously consider going that route myself.

There's no digital media that has proven to be completely reliable for very long periods of time. But I'm not sure that optical is much worse (if at all) than hard drives. I make my living as a still photographer shooting mainly digital. I have all image files on both hard drives and DVD-Rs (I just recently switched to Blu-ray). I have over 450 DVD-Rs burned with digital images. I've never had a single one fail (though I'm sure I will eventually), probably at least in part due to the fact that I take these precautions:

- use high quality media (authentic Taiyo-Yuden)
- burn at less than fastest speed
- don't write anywhere on disc (only center clear part with cd safe pen)
- don't store discs in sleeves--that can cause scratching. store them in DVD cases (I use slim double cases to avoid taking up too much space)

Hard drives are great, but eventually nearly every one will crash or get corrupted. Plus you can get a virus on hard drive or accidentally erase data; this can't happen on an optical disk (unless of course there's already a virus present before burning). I think the key is to have at least two backups on at least two different media types, with at least one stored offsite.

There's a good book and forum on digital asset management here:
The DAM Forum - Index

Hi Tom:

I have done the same but you should know that almost all optical media is prone to digital rot. I used to work for Circuit City back in the 1980s at the dawn of digital consumer audio, namely CDs. I own thousands of them. Approximately 5% of my collection has become oxidized, meaning that they basically rusted. They fade, then turn lighter orange, then eventually turn a deep red/orange and are unreadable. Some of these CDs were as recent as the late 1990s, some were made way back in the late 1980s.

DVDs are the same, except I have not seen the same oxidation, I have seen literal rot, where you hold the DVD up to a strong light source and you see holes in the substrate. The only optical media category I have not experienced digital rot or oxidation on are the Mobile Fidelity 24k gold CDs.

Hard drives are just as bad, I own a total of 168 hard drives. Most date from the dawn of digital consumer video with DV from late 1990s. I boot them on a regular basis to spin and lubricate the platter bearings and they are stored in a dark, climate controlled environment. On my last round of exercising, out of an attempt to boot 30 drives, 8 of them would not boot. I think the data is still on the platters, but the drives could only be rescued by a data recovery service, which costs a fortune.

I have had the least amount of long term storage degradation from tape. I still regularly watch VHS video tapes I recorded back in the 1980s. They have degraded since then but at least they will still play. I also have a lot of tapes from orphan formats like Hi-8, U-Matic, U-Matic SP and Beta SP, most of which still play decently.

I don't have any solution for you, but just beware that all of us are deluding ourselves if we think that any of the content we are generating today will be playable/recoverable in as little as 10 years from now. I sincerely believe that most of us in our old age will not have access to much media from our lives unless it was shot on film or possibly archived to a tape format like LTO or DLT, etc. And that is a big maybe since there will not be machines around to decode the data off of said tapes, even though the tapes may be intact.

Short term storage, we are fine but long term storage is not looking promising at all.

Dan

For a few years, in addition to keeping two copies on two separate hard disk drives, I also burned to two DVDs. Then I discontinued the HDD copies and went with two burned DVDs of every file. When I got up to about 1300 DVDs I realized that migrating that data from mollasses-slow discs to the next generation of storage media (not to mention accessing files quickly and easily) would be a nightmare. Reading from disc takes forever and forever... copying them all to the next media format would be hellish feeding of discs and waiting and waiting. In the meantime, large capacity HDDs got fairly inexpensive.

So now I've ditched the disc copies and I'm back to using HDDs (mostly 1TB) with two onsite copies and an offsite copy (or two!). The desktop toaster docks for running raw 3.5" SATA drives are awesome!

And when it comes time to transfer the data to another drive or the next storage media format, HDDs are a easy to connect and to read from. And day to day access and searching is so quick and easy.

It is KEY, however, as others have mentioned, to expect that all media/devices will fail sooner or later, and you have no idea when, so redundancy is vitally important. As is on and off-site copies of anything you don't want to lose. And smart to run all of the drives a little, now and then, to try to maintain the mechanisms.

Last year I decided not to buy any drives smaller than 1TB. Now I am starting to buy some 1.5TB and a couple of 2TB drives. But I still think 1TB drives are a good balance between capacity and risk.

I just got a SATA dock (among other things) that has a built in card reader from dealextreme for a really good deal and it only took two weeks to get here from china (i paid for the shipping).

I will just be buying 1TB+ drives and using the docking method similar to what Chris and Steeve are doing. Life is too short... I like the way you think Steeve.

One thing to remember when shopping for desktop docking bays and HDD enclosures... go for both eSATA and FW800. Use the eSATA when you can and for most Macs, FW800 is standard. FW400 and USB2 for connectivity are reaaaally slow.