Archiving XD Cam EX Footage
 

Just wondering if you guys/gals plan on archiving your raw footage as well as your completed work?

Depends
 

I think I will keep what I think is "worth it" and trash the rest. I thought about that recently as I usually send my clients the raw footage on Mini Dv tapes. I may give the option to put the raw footage on a separate DVD or optical disk "a la mode"

In the last few days I picked up a corpotate client that wants me to shoot in 4:3 SD !!! :( bla. then give them tapes for someone else to edit.

I guess if I was in EX land I would need to shoot and then give them a disk or drive for their house to finish. I really do hate tape but it is still nice and cheap for archiving and handing over.

MRW

I think best way to archive your EX - footage will be a XDCAM - Disc. The file system will be the same, so if you make a 1:1 copy from EX to disk you can play back this disc in any XDCAM-Deck.
Sony offers a XDCAM-USB Deck for less than $2000. This Deck has only a USB plug and no other I/O connections. Its for files only and supports 50GB-discs.
On the disc you also have a folder for any file-types you like. So you can put your projectfiles, images, music also on this disc. If you use the same file an folder hirachy on your NLE, you have your whole project saved 1:1 on a disc in your archive. You can change any title or grafic years after you finished this project. (when you use the same NLE - software ;))

greetings,
robert

I like the sound of that
 

That actually sounds really good. Are they re-writable? Tape or RW disks for the same price of the deck... hmmm

I'm glad I didn't invest in the HDV decks. It is a little bit of a PIA to go back to cam after having two nice panny (dv2000) decks but I just couldn't make myself do it.

It helps to have someone just lay out a solution. The disks are how much more or less?

MRW

The discs are re-writable up to and beyond 10000 times. They cost around $25-30 each for the single layer discs which are 23.5 gb. This is what the current XDCAM units use for recording media. Unlike a DVD, they are enclosed in a protective shell.

-gb-

In other words they'll only hold 1 x 16 gig card per disc?

(Aren't 32 gig cards supposed to be available by the end of the year?)

Cheers Vaughan

That would be true for the single layer discs. They now have the dual layer 50gb discs and the soon to be released stand alone drive (PDW-U1) will support dual layer discs. At initial launch, it will be a read-only drive for ingest and will have a later firmware update to be a writable drive unit.

-gb-

Or 3 x 8GB cards which make an almost perfect fit for the 25GB disc. You could even do 1 x 16GB card and 1 x 8GB card.

If you only use the 16GB cards then the second card I would just make sure to swap out before it was half full. That way you will know you are getting the most out of each disc.

I can only imagine how much the 50GB discs will cost. It may be much cheaper to use multiple 25GB discs. If the price will be anything like that of single layer DVD vs. dual layer DVD then I'm sure the price will be very high.

Even so, the cost per minute of recording time will still fall well below that of HDCAM tape.

As I noted in another discussion, the cost of hard drives has dropped to a point where you could store an hour of XDCAM HD footage for about $3, or $6 per hour with a redundant copy on a separate drive for security. Plus a single 500GB drive would hold about 30 hours of footage without taking up much physical space compared to a stack of XDCAM HD discs or even a stack of Blu-ray discs. Time required to archive to a hard drive is minimal and you can retrieve the footage directly for editing later. And instead of spending big bucks on a deck, spend that same money on another editing workstation, or a nice vacation...

OEM Hard drives / Blu Ray disk
 

Hard drives are the only viable option. For the cost of that XDCAM deck alone i could have shelf with 35 or more 250g hard drives.

The cheapest method of storage as of right now is buying OEM Hard drives and backing up your HD footage to them - then storing on a shelf. This is what i do with my HDV projects right now. You can get a USB to SATA ii cable - $30 - and just hot swap them out when you need something.

250g OEM drives can be had for $50 bucks. 500g for $110 or less.

This price beats Blu Ray disk at 25g for $15 and 50g for $30. Think - would you rather spend 30 bucks on 50gigs or 50 bucks on 250gigs. Blu Ray disk sounds like its the same cost per disk as XDCAM disks.

I dont see any reason to buy a XDCAM disk recorder for 2 grand if you dont have a camera that uses it - and then you have the cost of all the disks...

It's all what your individual comfort level is. I won't archive to hard drive because they spin at high speed and have low torque motors. If they sit on a shelf long enough and the motor encounters any resistance down the road, it might not spin up. You also have to rely on the electronics not to fail you (I'll admit that's pretty rare) vs. a disc in an enclosed cartridge.

The other purpose for the deck is to allow post houses and other facilities to readily accept XDCAM material without the expense of a true editing deck. For those folks, 2 or 3 grand is peanuts compared to what they've been having to shell out for DVCPRO HD or HDCAM decks. That's the beauty of file based media.

I suspect at the price point of the XDCAM EX, a lot of folks will consider hard drive back up to be acceptable. And who knows what other technology might come along in a decade. Heck, flash media may become the standard for archival.

-gb-

Any single archiving solution can't be considered infallible, so if you want security then duplicate backups are essential...and two hard drives are about as good as anything else. The odds of two drives both failing while sitting on a shelf are pretty slim.

That depends Kevin. If a drive has an inherent design flaw, say a grease that dries out out or component that absorbs moisture or breaks down over time it is likely that both drives would fail at a similar point in time. If they sit on a shelf for a year or two before being used you could very well find them both dead. Electronics like to be used, especially capacitors.
Much of this applies to tapes as well which can deteriorate quite quickly if not stored correctly, as for XDCAM disks, my guess is they will last very well and for that reason I have started to back up all my older valuable DV and HDV footage to XDCAM.

I agree with Alister. You would have to at least once a year hook up all your drives to check to make sure they still work. The older they get the more you will have to do this. The older the drives get the better the chances of the two backup drives not working either.

For short term storage such as less then five years hard drives may work very well. For long term however after five years the risk gets higher and higher of all drives failing.

As an example I recently put together an older system to use to experiment some of my own programs on. For this system I dusted off some older hard drives I kept packed in a box. Some of these drives were a little over five years old and sure enough some of them died right away. A few of the other ones died a few weeks after that. If the drives were a few years older perhaps none of them would have worked at all. Of course I do have one SCSI drive that is almost eight years old still working well but I do not trust it to anything important.

Of course that means actually hooking up all the drives to make sure they still work. How else will you know if one has died to quickly make a new backup of? Your shelf could be full of dozens of drives that have died and you may not even know it. By the time you go to use that drive it's backup may be dead as well and then you are SOL.

It's not the cost of hard drives that puts me off using them to store video, it's the constant checking them out and making new backups when they fail that freaks me out. I would have to eventually hire an IT guy just to check my drives all the time.

Fair enough, but isn't it also true that we don't know how reliable any of the recordable disc formats are going to be over time? I can see how even duplicate hard drives could go bad after sitting around for a while, but I wouldn't consider that any riskier than any single-copy archiving solution. The ideal thing would be duplicate backups on different types of media to avoid similar failure risks, but then that's a pain to manage.

By the way, a rigorous data protection plan would call for having multiple copies at physically separate locations, for what that's worth.

I agree that we probably do not in reality know how long XDCAM (or any other recent format) disks will last. They should be good for 50 years, but by then XDCAM drives are probably going to be pretty rare! The beauty of XDCAM disks is that it is just the disk in a cartridge. The mechanics and electronics are in the drive, if the drive fails you can get another one and nothing is lost. XDCAM disks are waterproof and un affected by humidity so even if your home or office floods you footage should be safe. They can also stand high temperatures and cold, so all in all a very robust medium. I keep working copies of my material on hard drives with backups on XDCAM disks stored in separate locations. My most valuable footage is also duplicated and held by 3rd parties for peace of mind.

Also if we are just talking about raw data then don't discount other optical technologies that when they become fully main stream will be less expensive than a niche product like XDCAM.

A dual layer BlueRay disk will hold 50gig. Once fully main stream and low cost it will be a viable choice for backup of video files for the compressed formats including XDCAM.

Chris

Yeah but by the time writeable Blu-Rays become common and cheap we'll be at $150 per Terabyte with hard disk drives. The march of cost/GB of hard drives will continue to obliterate other options. Even if you replace your hard drives every 6-7 years or so remember that by that time the capacity of the drives will have increased and price will have dropped significantly. Just think back to 2000 and what you thought was a "large" hard disk then is a joke now. Also, because the capacity of hard drives is so much greater than individual optical discs, it will also be a lot easier to manage.

So I'd go with hard drives for archiving and just plan to replace them periodically. Also get a hot swappable drive enclosure to make life easier.

Not quite the same. Consumer Blu_ray doesn't have all the error correction and robust file system, or the dust sealing of the XDCAM discs.

Agree with multiple back-ups. But here's another little item I just remembered from my days as a systems builder. As you know, the drives have a low level format and high level format. The low level format was the drive and the controller being hitched up together. The high level format was what the OS wanted to do (FDISK on DOS/Windows). I found that drives which sat around for extended periods would develop read errors. It was explained to me that the low level formatting information can become corrupted due to the ever so slow loss of the magnetic charges on the disc.

Granted, this was many years ago so maybe the retenticity has improved over the years. But it was a potential liability then. The solution was to back up the drive annually and go do a low level format to refresh that part of the equation. But these were 20 mb drives and it didn't take all that long. BTW, here's the trivia portion. Most controllers had their low level format routine started by opening Debug and entering G=C800:5.

-gb-

Bigger and bigger drives means more and more material will be lost because of a single failure. I would never fully trust a hard drive. I've had too many fail on me, I have a large box of dead HD's, many of them only 2 or 3 years old.

I want to put my footage on a medium I can put away in a box and forget about until I need it. That's the way it is with my tape and XDCAM disk footage. I don't want to keep dumping from one HD to another HD, that's how footage gets lost or corrupted. If you copy 500 gigs of footage from one drive to another are you going to go through every file to make sure none of them are corrupted each time? It happens, and every time you have to make another copy you risk adding more errors or loosing more files.

That's why Banks and others with crucial data still use tape for backups and companies like EXABYTE are still in business making tape backup systems. Failures are rarely catastrophic with tape, unlike hard drives.

I suppose it all depends on your risk/cost calculation goes. Yes, I've had hard drives fail. But that's why you'd backup to two drives. I'd never suggest a backup system that relied on one hard disk drive. I still believe the likelihood of two drives failing within 6 or 7 years is so small it's not worth worrying about. As far as corruption or losing files during transfer goes, if you do a verified backup it virtually eliminates that risk.

Magnetic tape also fails pretty frequently in about 20 years. I have played Betacam tapes from the early 90s that clog the heads within 30 seconds of playback making them virtually useless.

Yeah, the actual "rated" lifespan of magnetic tape is pretty short. You'll see dropouts (or at least lossy drop-out compensation) on DV tapes within 5 years - unless you have a 5°C 20%RH storage facility, of course!

Give me hard drives or optical media any day!

Tape backup for video
 

This may not be a solution for everyone, but a few years ago I bought a Quantum LTO-2 tape backup (the successor to DLT). There is now LTO-3 which is even faster. It backs up onto a 200GB tape at about 1 GB per minute. The tape cartridges are only $34.00 (the unit is $1,200.00) and the tapes have a 30+ years life span. LTO is used by most Fortune 500 companies for reliable data backup. An Atto SCSI card is about $300.00-450.00. This really works well for me.

I went to the LA FCP users group meeting in January (2007) at MacWorld. There were two guys who had just finished editing the film Zodiac for Paramount. They were capturing directly onto hard drives via FCP and backing up onto LTO-3. At the same event the Panasonic rep showed everyone the new DVCPRO HD deck. It was a mere $25,000.00. Just think about it. We’ll never have to purchase another expensive deck again, never have to worry about tape dropout, etc.

What I'm reading more and more though is, as others have said here, that HDDs need to be spun up/accessed occasionally to keep it in proper working order. Though, if you only have a few external HDDs in enclosures then having a "spin up day" worked into the calendar a few times a year shouldn't be to big a deal. I'd still be ready to rotate all the data onto new storage devices or mediums (depending on what's come out) every 5 years or so though.

Me personally, I'm looking into tape options like Dave Bingham talked about.


-A

Hmm, the LTO-3 seems rather interesting.

Now can you back up the full HD (1920) file or does it get compressed to 1440 rez when transferred to tape for back-up?

Why not transfer as data? Completely format-independent.

Hmmm, didn't know that was possible with tape.

I have never transfered Data to Tape as a back up... other than the tape used for capturing footage in my camera of course.

Archiving
 

FWIW, most of the people who attended the recent High Definition Summit in Los Angeles use LTO3s for their back up systems. (we're talking networks and high end production entities here, mostly in sports and other high memory / bulk storage arenas)

They were also talking about wavelet compression based technology (redcode, pro res, jpeg 2000 etc) and so-called intermediate mezzanine formats, and the current "best method" seems to be a dual one of HDD RAIDs plus the LTO3s.

However, the times, they are a-changing, and much faster than before. The newest compression technologies could mean emailing a movie, or near-VOD in full HD over the internet, within a year.

So a reasonable solution for the smaller operation may well be LTO2 technology. It's not the latest flavor of the month, is very well rated for long term storage, and the $1k plus LTO3 drives will read the media.

Trouble is that most of the LTO drives I've seen are SCSI devices...

Here we go again :-)

In my old days with Unix workstations some 15-20 years ago, SCSI tape streamers were the only backup choice...

Tape is still the main storage for data in larger companies....

Out of curiosity, how much does it cost to buy a drive which can read/write XDCAM discs?

P.S. It seems funny that we're all excited about finally getting cameras which record to something other than tape, then talking about backing up to tape... :-)

Very interesting Piotr. I appreciate you offering this information. It seems like this is the route I will go when I get my EX1 early next year!

I remember reading a while back about digital VHS as a data storage medium.
Could that have any relevance here?

Kevin - I agree, there's considerable irony in welcoming tapeless acquisition then using tape for back up ...

Does anyone have experience with iomega's REV70?

I'm comparing mac-friendly archiving solutions.

I don't think hard drives are a good long term on-the-shelf archive solution.

LTO drives are crazy expensive and most seem to be SCSI but the media is only about .12 a gig for LTO-3. Plus, I'm having trouble finding a mac-friendly LTO drive... anyone?

The drive for Sony's U1 is $2800 and the media costs about 1.40 per gig (and for now, the firmware doesn't support the EX1's full raster). Plus, I think it will only back up video files. It's not a solution for backing up everything else associated with a project.

The REV70 is 70 gigs, the drive is under $400 but the media is about .85 a gig. Plus you can use the REV70 for backing up other data, unlike the U1.

So unless I can find a mac-friendly, affordable LTO solution, I think I'm going with the REV70 firewire drive.

opinions?

SCSI still available
 

Chris, as far as have seen lately on the net there are quite a few SCSI cards für Desktop Macs as well as for Powerbooks or MacBooks Pro for that matter.
Unfortunately I can't tell you how reliable these work but if you buy an LTO drive from a system dealer ask him to provide the SCSI cards that fits. If it does not you can give back the whole bunch of useless gear. If its fits, be happy :-)

As long as Sony offers their XDCAM drives and similar expensive solutions, don't expect them to come up with something with a lower price. If you want a lower priced solution, go to the IT people and look how big companies make secure backups that saveguard data at least decade or even longer (as required by law). There is no cheap and at the same time secure data backup.

BTW, Piotr, with digital video, all the workflow is about data, nothing else.

Sure it is Peter - I was just trying to realize that to Brad...