DVCPRO HD storage for masters
 

Today I woke up a little scared about something I haven't thought about...
If don't own a DVCPRO HD deck, how will I archive my DVCPROHD masters?
Will 4.7 GB DVDs will give me just 40 seconds of 100 MBs video? or will we be able to store it in the same format that the P2 achive it so we can have like 4 minutes per 4GB in 1080i or 8 minutes in 720p?

How much time will we be able to put as data on a HD-DVD or Blue-Ray?

Will it make sense to master in HDV?
-EDWIN

HDV is not a master quality format, so I'd avoid that. Your calculation for amount of DVCproHD on DVD is wrong, you'll get about the same length as you'd get on a 4GB P2 card, which is between 4 and 10 minutes, depending on frame rate.

Graeme

If I can get nearly 10 minutes of a 24p 720p recording on a single 4.7GB DVD, I'd be really satisfied.
-EDWIN

You can get 10 mins on a 4GB card, so you should be able to get 10mins on a 4.7GB DVD without any bother at all. I've got the feeling that for the moment, DVD is what I'll be using too.

Graeme

On a regular DVD disc (4.37 actual GB), you can count on 4.3GB for file system overhead. In terms of 100Mbps DVCProHD, that's 5.87 minutes of storage (1080i or 1080p24 or 720p60). For 720p24, which fits into a 40Mbps stream, that's 14.67 minutes of video storage on a DVD. DVCPro50 stored on DVD will be 11.74 minutes.

If you write to a dual-layer disc (DVD9), you can double all the above figures.

DVD may be the most cost-effective back-up media for many at first. IMO, DLT makes more sense in terms of capacity and if you will be storing a lot, then it will probably pay for itself in the long run. However, DVD is still a more reliable storage medium than yet another magnetic tape format, which DLT is. I would imagine that within 18 months, we will have Blu-Ray drives and 50GB RW discs and then backing up our DVCProHD100 clips will be of minor concern. A 50GB HD-DVD or Blu-Ray disc will hold over an hour of 100Mbps video. Multiple layer writable discs should follow within a year or two and capacities are expected to top out at 250 to 500 GB.

Yep. And HDV is only a 25Mbps format, so it theoretically can only deliver 25% of the quality of DVCProHD. We'll have to wait for the HVX200 to see what it can do, but comparing DVCProHD shot with the Varicam to the current crop of HDV offerings, there's no contest... HDV sucks.

DVCProHD is 100 Megabits per second (at full data rate), not 100 Megabytes, so your calculation was off by a factor of 8. At 100 Mbps you should be able to fit up to 6 minutes of 1080i video on a 4.7 GB DVD, or 11 minutes on a dual-layer disc. With Toshiba HD-DVDs you'll get 20 minutes on a 15 GB single-layer disc; with Blu-ray it'll be 33 minutes on a 25 GB disc.

Don't forget to plan on making two copies of everything if you want to be safe about preserving your master data.

how did you come to this conclusion?

It's not that simple because the codecs don't work the same way. Also note that DVCProHD has fewer recorded pixels per frame at 1080i resolution, so that's a trade-off in terms of image quality. Overall DVCProHD should yield better image quality, but nowhere near four times the perceived quality.

Exactly. And to see how final image quality compares when delivered to customers and viewed on typical HDTVs.

That's funny considering the difference in camera cost, which obviously has bearing on the image quality beyond the difference in formats. We all know that DVCProHD is a more robust recording format, but HDV is surprisingly good for the price and will be the de facto standard for affordable HD production for the next few years.

http://hdvplus.net/page22/page22.html

Over 15 years of experience with both magnetic and optical media types. Not to mention, most every media manufacturer claims their optical products are more reliable and have a longer lifespan vs. their magnetic products. Don't get me wrong, magnetic media is still very robust, especially the more industrial types like DLT. However, it is still vulnerable to magnetic fields and the fact that tape components involve moving parts. Optical media in the form of CD-R/DVD-R has been shown to have a shelf life of over 15 years and manufacturers claim up to 100 years. Obviously, there's no way of proving that since 15 years is about as old as optical formats get. But you won't get that kind of claim for magnetic media and I know that I start seeing issues with old tape masters (DLT, DV, Travan and otherwise) after they have been around for 5 years or more. The very nature of magnetic tape allows for the magnetic polarization of the tape coating to bleed and crawl over time. Optical media has its flaws too... Heat, UV or sunlight or other radiation can still alter the writable film in these discs. But my experience reflects the current claims of most media manufacturers and I'll stick by it.

My plan is to use hot swap sata hard drives for everything from transfer and editing (RAID), to archival (one on-site, one off-site), until Blu-Ray (Preferred), or HD-DVD (I'll take what I can get) gets established as commonly as DVD-R/DVD+R.

If I keep everything in 720p 24fps, the file sizes won't be too obscene. About 24GB per hour of footage.

So that would be 16 hours of archival space on a 400GB hard drive that as of right now that market price is about $250.

My productions will primarily be short films and music videos, so final masters won't be huge. I think it might take awhile to fill a 400GB drive with 16 hours of final compelling engaging worthwhile high quality creations.

Now I know everyone has this retort - "What about archiving the master raw footage dude!?!?!" Well, it could be time to scrap the idea of keeping all that dead weight around. I almost *never* go back to the raw footage. After 15 years of audio and video recording, my library is so massive, I notice this one thing. What I value most over time is the finished product, so I can present it and share it with other people. Painters have it good that they don't leave junk behind. The work is made and displayed, with no working pieces left behind. And I think I'd like to adopt this same philosophy in this psycho digital media world.

If I did think it was crucial to keep raw source video and project files, I guess I gotta just buy more hard drives. :)

Also, think of this added benefit. If you need footage or masters from your archive, you can retrieve it FAST with hard drives, by cataloging everything with cataloging software. Although on the pricey side, this system blows tape away - far away.

Whew. Now, when is that damn HVX200 gonna arrive at my door?

steev
http://www.holyzoo.com

Jeff,
I have a bunch of 8mm video tapes which are close to 12 years old. They contain priceless family footage that I don't want to risk losing. You think they have already started to degrade considerably? I've always kept them safe in a dry, airtight box. I don't have nearly enough drive space to digitize all of them right now. I'll probably have to wait for HD-DVD's. That brings up another issue... I wonder how vulnerable HD-DVD's will be for archiving? It appears that one of the formats has a protective case for the disk. I also wonder if the disk coating durability has improved. The 15 year shelf life you mentioned for regular DVD's is scary. I'm curious what it actually is in a protected environment. I amount of money I've spent to build my DVD movie collection makes me nauseous.

Steev,

Your thread made me laugh. I'm so guilty of holding onto source footage that I haven't touched for years. I also have this irrational fear of removing archived footage from hard drives to only remain on CD's/DVD's. I've had amazing luck over the years with hard drive dependability (I just jinxed myself).
I'd love to have your attitude... finish the project and cut everything else loose. I think about deleting the source files, then things like color correction, re-editing, recompression pop in my head. I think I need therapy. Just can't let goooooooooooooooooo!

Damon, I am a major media packrat, so I totally know your pain! What I found this last year after consolidating CDs to DVDs, is that it wouldn't be the last time. And I hope it isn't the last time, because like other things in life, we need to purge, reorganize, and renew. This is leaning away from the topic, but it's related, because the HD revolution really brings up massive questions and problems regarding storage and archival. At NAB, it was something that had nearly everybody worried and scratching their heads, while companies were lighting up because that means big $$ to be made in storage and media management, etc.

Meanwhile...

I'm trying to think differently. :)

Finally, regarding archival, like your family footage. I obviously wouldn't say "just scrap it man!". I'd say, put it on redundant hard drives. :) One copy at home, one in an offsite secure location. If you were really paranoid, add this to the equation. Take big files and make segmented .dmg files or .zip files and burn those to redundant DVDs - one on-site, one off-site. Starts sounding like a full time job, don't it?

The segmented .dmg technique could be an option, with HVX200 footage, for the extremely patient.

steev
http://www.holyzoo.com

Another video packrat here (raises hand). Archiving/backup is a big issue, especially with the advent of HD. I have a lot of useable stock video and If I lost all of my tapes to a fire or other catastrophe, it would be arguably as, if not more, devastating than losing all of my gear. Gear can be replaced, years of video work can't. DVDs are attractive as they are non-magnetic, but are also very sensitive to scratches.

At the end of each year I go through all of my tapes from that year (usually over 100) and capture in anything that I feel is worth hanging onto for future use. Most of the worthwhile material from the past year is already on the hard drives anyway so this isn't as daunting as it may seem. I then condense all of this into compact 'packages' and export. I take all that video and save it digitally in three places - two external hard drives and a third sent to a friend in another state.

Then, I export everything to tape four times, so I have four complete sets on tape (usually 4 sets of 6 to 8 tapes). I keep one set at the office, take one set to safe deposit at the bank, then take the other two to out-of-state family for storage.

That way (barring a nuclear holocaust) I should have at least *one* intact set of tapes left somewhere.

When DVD gets to the point where I can affordably fit all of my archives to just a few disks, then I'll make that jump. But for now the redundant tape and hard drive backups give me peace of mind.

Another option I've considered is to rent a deck once or twice a year to make master copies of anything valuable.

One vote for Dan as the Most Organized Pakrat of the Year!

Dude, your diligence in actually organizing, backing up, and safely archiving your footage (not just saying you should get around to it, like me), forces me to hang my head in shame.

My poorly executed plan is to:
- keep all raw footage tapes
- leave anything I might want to use on the hard drive
- back up key stuff to DVD-R, including complex processing intermediates (quicker and easier to save the files than recapture from tape)
- back up all but the really useless raw footage and final products to tape

Sadly, most of that is still on my to-do list...after barely dodging Hurricane Rita, archiving all our photos and videos HAS moved up the list, though!

I know that most people think CD's or DVD's are a better way to archive over any form of tape. And I do think the same except for one thing: I've seen older discs that all of a sudden are not recognized by a newer computers, or whose files are now not supported or corrupted. It happened to me with some photographs that were scanned and everytime I try to open them, they just don't. In the case of tape, maybe not with the same quality, but you can save a portion or some of it.

Also, optical media has gone sooooo cheap, that it is hard for the bad brand to go any cheaper than the good brand and all of a sudden you get confused on which one to buy, you buy the one available because it costs almost the same as the good one and there you go... trusting your work to a $0.60 disc!

So... how reliable are optical media nowadays?

I vote for Dan as well. I try for triple backup, but damn man, you are the master at mastering. :P

I know this may sound ridiculous, but the crazy part about all of this is to attempt to archive, to hold onto things "forever". Meanwhile, I'm starting to get the real odd sense that the lifetime is flying by real fast and soon it's really not going to matter anymore. From there, it will be up to our ancestors/archivists to either hang onto our work, figure out how to decipher, use, repurpose, present our work, or just junk it.

So the question remains, what is a viable working solution for the present day?

DVDs, Hard Drives, and Tape.

I haven't had many CDs or DVDs fail me. When I converted my CDs this year, the oldest one was from 1998 I believe. So that's 7 year reliability. If you have redundant copies, I think you're pretty safe there.

Hard Drives, I have less experience leaving them alone for years, then cranking them up and using them. Anyone with more experience with this?

Regarding tape, we all know tape can really get screwed up over time, magnetic field hashing, signal loss, tape surface sticking, and with one bad slip on a deck, the old "this thing ate my f'n tape!!" I think tape is solid if stored correctly (at a former job, they had the tape archive around where the janitor would come by twice a week and neerrooom past with the vacuum!). But I am intrigued by Dan's notion of renting a deck and archiving once or twice a year. Great idea, and free up the hard drives space.

...Simple laws of physics - yes, they have degraded. How far gone they are will depend on lots of factors like the environment they were stored in. Airtight boxes are one thing, but there's also the consideration of temperature variations, electromagnetic fields and radio waves. Even what the tapes themselves are made of... Some of the real old tapes in my archives even show signs of the plastic case and tape material starting to break down. I've got some old 8mm and beta tapes as well as some audio cassettes that had clear plastic shells on them that have really turned yellow now that they are 15~20+ years old. Even being stored in a cool, dry environment away from light.

If you have taken care of your tapes, they should last a considerable amount of time, but I would be a little nervous about magnetic media of that age. You may want to take a look at what's on those tapes and see for yourself... It may be time to get that video archived digitally before its gone forever.

HD-DVD will have the same shelf life of current DVD-R/RW as it will be constructed of the same materials and will just carry a much smoother substrate surface for the laser to write to. Writable Blu-Ray media is still somewhat of a mystery. 25GB Blu-Ray drives are seeing limited availablilty in some Japanese markets, they have been for nearly a year now. Sony has been beta testing them with larger corporations and educational institutions. They claim 100+ years the shelf-life of their media.

Shelf life of current CD and DVD media is estimated to be 100 years or more (both for the writeable -R and +R variants as well as the manufactured disc types. The 15 year lifespan I mentioned is that's all anyone can confirm in reality at this point for CD-R/DVD-R. CD-Rs have only been around for about 15 years and they really didn't become a consumer product until about 10 years ago. We won't know for at least another 80 years or so whether or not our current CD/DVD media really has a 100 year shelf-life, will we? Laboratory simulations and stress testing conducted by the manufacturer can only give us an estimate. These manufacturers are all claiming this 100+ year shelf-life for disc media even though several of them use different types of dyes and films to create their discs... None of them really offer an ironclad guaranty either. In 100 years are people going to file suit against DVD-R manufacturers because great-grandpa's DVD home movies don't play anymore? Doubtful... 100 years from now, DVD will be so obsolete that if any still exist in working order, they will be relics in an antique shop. The reality of digital storage solutions is that an archive of your digital video or other data should continuously evolve with the available standards. I have gone through all the work to convert all my analog video into digital form. Even the old 8mm and other formats of filmstrips shot by my grandparents when I was still a wee lad.

But what we do know for sure right now is that CD-R and apparently DVD-R can be stored at least 15~20 years without worry and the manufacturers claim 100+ years. You won't see any such claims for a magnetic tape media. Tape media consists of a sprayed-on metallic coating (that's way thinner than a human hair) on a plastic film holding a magnetic charge. The odds of this holding that charge stable for extended periods of time (10 to 20 years or more) is highly unlikely. Magnetic charges on microscopic particles will tend to shift, bleed and crawl over time. The magnetic charge or polarization of the particles in these coatings can also dissipate. Tapes, if properly stored, should last at least 10 years... 20 years is possible, but on an analog recording like your old 8mm tapes, I will be very surprised if after 20 years you don't see a noticeable amount of signal decay.

Yes, I simlified or generalized my statement too much. The perceived quality level of HDV is going to be a lot closer to DVCProHD than what the bitrates want to indicate. But I was attempting to look at it from an overall information point of view... 100Mbps vs. 25Mbps still allows for 4x the data flow when comparing DVCProHD to HDV. In some ways DVCProHD isn't as efficeint as MPEG2 and that helps bridge the gap between the two formats somewhat. HDV does allow for more pixels, but also sacrifices color information. DVCProHD will provide a lot more information to work with in terms of compositing or other post production tasks. I don't expect the HVX200 to rival the Varicam in total image quality, the lower cost fixed lens and smaller sensor block probably won't allow for that. But I also expect the HVX200 to knock the socks off of any HDV solution out there right now or in the near future. Canon's XLH1 will probably provide a superior solution for certain tasks, but to take full advantage of that camera, you'll need an external deck as opposed to just recording to HDV.

I can't render HDV back to HDV even when there's no filters applied without seeing artifacts increase. DVCproHD to HDV would stand up better than this, but once you've gone down that low bitrate MPEG2 path, it's very hard to come back.

Graeme

The value of an Archive
 

Well, I can lend a little perspective to this discussion since I am in the archive business.

Funny that someone brought up painters- there is a major show of Van Gogh's drawings at the Met right now and they are worth quite a bundle. Of course he didn't see any $, but we get to see some amazing examples of a working process. Another interesting thing to me is that B&W pics of my grandfather when he was a infant are better preserved than the Sears Portrait Color pics of my sister and I from the 60's (if Sears had done Dye Transfer we'd be ok). Most of them are almost entirely faded and if the technology hadn't come along when it did they would have been totally gone in a few short years.

I've also worked in the News Biz and we are now making money off our archives. Trouble is you cannot sell footage from a pre-produced package because of duration of the shots and change in editorial POV. The increase in the amount of production companies and outlets for content are driving footage sales right now and clients are looking for high quality images that also don't look overproduced, so your b-roll can potentially be very valuable. I would save it if at all possible.

Dan has it covered about as well it can be right now and from my perspective he has the potential to make as much money with his outtake material if he owns the rights to it as he does from the final product.

Just something to think about...

We'll see, we'll see. We'd certainly hope it will be better, but since it's still a 1/3" sensor with (apparently) 720 pixel vertical resolution and probably a comparable lens to current HDV cameras, I'm not sure the difference will be all that significant. Color depth should be better and artifacts should be fewer, with better ability to handle motion, but will that translate to significantly better quality by the time it gets to the viewer's screen? Can't wait to see some video samples...

Even 720p without pixel shift vertically is the equal of 1080i in resolution due to the interlace factor to stop 1080i flickering. It is thought that the cam will use both horizontal and vertical pixel shift, although this is not confirmed by any means. Having the largest pixels of the bunch should make for better contrast / noise / low light, and that will be visible, but we've got to wait and see what design decisions were actually made.

Graeme

In the past year I have had 35 to 40 year old super8 film converted to miniDV for editing and saving to DVD. These old films survived quite well. If there is any doubt about the survival of your 8mm video tapes I would think it worth getting them converted to (at least) miniDV. I used a professional service to do mine.

One angle not covered in this thread, but which is perhaps worth mentioning is that in my experience there is a huge difference between the archival quality of a cheap DVD burnt on a cheap burner compared to quality media burnt on a good burner.

I have got a Lite-On drive which I use for its bitsetting abilities, but the burns it produces (even with good quality media) are seldom of archival quality.

For archival stuff I stick to my Pioneer drives and I test all the burns with Nero DVD Speed (though DVDInfoPro or KProbe will also do the job). A few get rejected, but I do then have reasonable confidence in the discs that are left.

How about a glass master?
Instead of just "burning" it into a polycarbonate DVD, would it be more reliable to take it to a replication facility and have them make a glass master, as if you were going to replicate discs (but of course you wouldn't have to replicate any)?

-EDWIN

But then the JVC HD100 also has a 720p sensor and recording, so any advantages to that should come across there as well. Point being that it's not clear the HVX200 will "knock the socks off" of HDV, but we'll find out soon enough. If I was prepared to spend close to $10K for an HD camera, I might be inclined to wait for next year's models with larger sensors.

When shot in a very controlled manner, the JVC can make for a very nice image, but the stock manual lens is rather poor and holds back the quality of the camera, and the HDV compression really doesn't show the pictures it can make in it's best light. I'd expect the HVX to be at least as good in controlled conditions and noticibly less compromised in more demanding conditions - and if not, I won't be buying one. Should be getting a HVX soon for testing....

Graeme

Graeme, Looking forward to you sharing your thoughts on the HVX200!
I sure hope it lives up to some or most of the hype.

Steve

Hopefully. But remember, I'm fussy. I do think that DVCproHD is too compressed, but from my experience, it's less visually compressed than HDV, but that's comparing Varicam to HDV. Hopefully the same or similar improvements will be seen with the HVX over the HDV. However, I still think that the SDX900 (from what I've seen), gives you the best image from a Panasonic camera, HD or not!

Graeme

Graeme: out of curiosity, what do you think are the best DV cameras in the ~$4-6K price range?

I have no idea. And I don't know what your priorities in a camera are. From the images I've seen the best affordable DV camera is the DVX100. If you want real widescreen, the XL2 or the Sony HDV in DV mode are your options.

Graeme

Thanks, that's consistent with what I've concluded that the Sony FX1 is a good value in a true widescreen camera. As far as that's concerned, the HVX200 running in DVCPro50 mode could be an interesting compromise for producing widescreen SD output. By the way, I did some tests recently which convinced me that FX1 HDV footage downsampled after editing to widescreen SD looks better than FX1 footage shot in widescreen DV mode: http://www.videomem.com/HDV/encoding...comparison.htm

I'm still debating how I want to archive my HDV footage, but hopefully next year's HD DVD options will help with that.

The BBC primarily use the Z1 as a high quality widescreen SD DV camera.

Downsampling HDV to uncompressed SD should look better than HDV downsampled to DV (as long as you avoid MPEG2 provoking situations) or shot as DV.

Graeme