Documentary Archive solution needed!
 

HELP…please!
I have been unsuccessfully thrashing around in the dark for the past few months trying to come up with the right solution to my production problem:
I am scheduled to begin shooting a documentary series in July next year. The entire series will be shot on and around a small 33 foot boat sailing through the northern and southern Pacific.
As you are probably aware, pressure is being put on producers by broadcasters like Discovery, Travel and National Geographic to shoot on film or HD. The conditions under which I will be shooting this series preclude me from using a DVW or other full size type EFP unit and film is out of the question. I need the flexibility and maneuverability of a smaller "handicam. So it's down to Sony or Panasonic…with Canon a possibility.
Here's where it gets difficult… I don't believe that Sony's HDV 1080i format will compare with Panasonic's 1080/30p (or even 720/60p) format. Especially given the extreme camera movement and complex in-frame movement some of the wind and sea conditions will cause. I need clean pictures with the ability to do slo-mo at times.
The big problem with Panasonic seems to be archiving the copious amounts of material I will be shooting. Don't forget that I will be on a small boat with limited space and a most hostile environment for electronics. From this perspective Sony and Canon definitely have the edge by being able to record onto tape, but how does the HDV codec stack up to DVCPro HD?
If I was able to select (pre-edit) only usable the usable shots and dump the NFG takes, this should reduce the amount of storage space I would require. Would I need an editing platform, or is there some way to do this without going through an editor?
Any input or suggestions or ideas will be happily received!
Cheers,
Bruce

Got electricity in the boat?

Until Blu-Ray DVD's become a commodity and fall in price like DVD-R's have, the best bang for the buck in archiving has to be external firewire or USB2 hard drives.

With clients like Discovery and National Geographic, I think you're right in leaning towards the HVX-200. The HDV codec is going to be much less forgiving in post, when you try to color-correct all the footage to give it that highly polished look that both those networks demand. And the HVX is the ONLY handycam-size camcorder that will allow you to do TRUE slo-mo in the camera.

Have you considered shooting 720/24P, then using 720/48P or /60P for slo-mo? That's the combination that will most closely emulate film, while also allowing slo-mo.

It sounds like what you'll be wanting is to use the 200 with a Firestore recorder, then once or twice a day dump footage through a laptop onto an external hard drive. Will you have access to at least 1 A/C outlet on the boat? You could run the laptop off batteries, you'll just need 1 outlet for the external drive, and then only when you're archiving the footage, at filecopy speeds (several X real-time).

Of course, the possibility was also mentioned of dumping footage off P2 cards directly to a USB2 external drive. I'm assuming, being a docu, that you're going to want longer run times than P2 will provide, therefore the Firestore. I wonder if you could dump from a Firestore, through the camera to external USB2 - no laptop needed! Probably not, but it's worth mentioning.

At less than $1/GB, hard drives are where it's at, at least for the next 6-12 months. I might also want to take the extra step of then cloning the contents of each drive to a second backup drive after each days shooting.

You can do that in-camera, instantly. The P2 system lets you review clips and delete bad ones, and protect good ones. Using that system you can keep your usage of the cards quite efficient.

Another thing to consider would be the newly-announced FireStore... at 100gb, it should provide for over four hours of continuous 720/24p recording (note: that's not verified yet, but I'm basing that off the hopefully-reasonable assumption that the drive will drop 'duplicate frames' and only store the active frames). I'm not sure, but I'm betting that you'd be able to delete bad clips from the Firestore in the field as well.

I agree with the above. May I add that, if you have enough room, it would be real handy to have an editor with you who can take the FireStore or P2s back and forth from camera to laptop/hardrives. And, in the meanwhile, this person can start cutting and throwing away unwanted footage, clearing space in the hardisks.

Just concerned about the loss in quality when the time comes to transfer up to 1080 from 720. Also thought that the occasional frenetic movement of camera and frame content would be better handled by 30P. Could I not mix the two formats? ie: 1080/30P for majority of recording while 720/60P for in-camera slo-mo?

Power will be available via an inverter from a large battery bank so laptop and external hard drives are certainly possible. My concern is the number of hard drives I would need to archive +/- 100 hours of HD material! This is a series I'm shooting, not a single one-hour documentary. Sorry if I didn't make that point earlier.

That would work well. From an editing point of view it would certainly streamline the process - something we should be looking at with this new type of technology. (I'm an old f@rt - even remember working with 2" videotape!;)

No space...no budget...no need. I will be "out there" for a year or two and will have the time to do all the grunt work myself. My only issue would be when I'm shooting ashore and have limited storage and carrying capacity - no SUVs I'm afraid, just Shanks' pony!

Thank you all for your input. Fortunately I have a few months before I leave next July.

Cheers,
Bruce

It's been discussed many times that folks would shoot 1080 for the bulk of the production, and only use 720 for the off-speed shots. I have two concerns, neither of which will be addressed until we actually see the camera.

1. Since the CCD's are almost certainly not full 1080 native (which is actually a good thing), there is some sort of interpolation going on to get a 1080p image, most probably having to do with pixel shift. How good is it? who knows? Perhaps there is NO discernable benefit to shooting 1080p, as opposed to 720p with up-rezzing.

2. 720 and 1080 may not intercut seamlessly, even from the same camera, because of the above.

Panasonic has been pretty bold in talking up this camera as a b-cam to a Varicam shoot. Have you seen Varicam footage? IMHO, it's plenty of quality for a Discovery HD program.

Also, are there any PAL markets for this program? If so, 24p is the only framerate that can be used in both NTSC and PAL territories. At 30p, you're pretty much locked into NTSC. I guess it all depends on what the broadcaster is looking for.

Also, I don't think that 15-30 external HDD's (by Barry's calc., with overhead - 15 w/o backup, 30 with backup) at a total cost of $7-15,000 is extreme at all for storing 100 hours of HDCam. In fact, that's as cheap as it's been, EVER.

Seriously, go for tape.

If you're going to be out on a boat for a year, do not mess around with hard drives, lap-tops, etc. It's just begging for trouble. The HVX-200 while being a fine camera (on paper, no one's used it yet), doesn't look like a good choice for documentary work where you have hours and hours of footage.

The workflow is not there yet for long form documentary work, requiring the archiving of so much material. However perhaps by your shoot next july there will be a solution that makes sense.

Another good thing about mini-dv tape is that it's ubiquitous. I once bought a 5 pack of sony dv tapes literally ON the mekong river, from a guy in a longtail boat. If you're travelling around from port to port, there's always somewhere you can buy DV tape. An extra USB hard drive may be more difficult to find.

I will agree that the HVX does have a much better codec than HDV cams, but will that alone result in a better image? The HVX isn't out yet, so no one is sure about the lens and CCDs. Codec is not the be-all and end-all factor in the image.

If the choice is a slightly better image, pitted against the chance of running out of HD space, or heaven forbid catastrophic hard drive failure. It's infinitely better to have a slightly more compressed image on tape, than no image at all.

Still by the time you go on your shoot, the situation may have changed. July is still a long way off. So my suggestion is don't buy now, wait till all the cameras (Sony, JVC, Panasonic, and Cannon), have been out for a while and time tested in the field before you make your choice. This board is the perfect place. If there is any little flaw in a camera, rest assured you will hear about it here.

You may find this BBC Guide to HD helpful. It is still in draft, I believe, and contains some errors. It is not yet supposed to be available to the public but someone found a way in. It provides advice for those new to HD production and direction. Link:
http://www.bbc.co.uk/guidelines/deli...Bookv01_00.pdf

I agree 100% with John. You cant do selects on a documentary until you are done shooting. Also, you need to have everything archived and backed up. Do you really want ALL your shots on a couple hard drives in an environment that is uncontrolled? Right now there is no solution but tape for docs of this type and the only affordable and portable HD solution is HDV. I am not a fan of HDV but it would work best for you if you have to go HD.

This is the big problem with both the HVX and the XLH1. To get the best HD results you have to spend a lot more money and it becomes less portable. I personally wont invest in HD gear until there is a more solid and affordable collection media.


ash =o)

I would be very careful to ask the prospective networks if they will accept HDV shot material. The choice of networks leads me to the opinion that the highest level of quality is desirable. I would bet the HVX-200 @ 720/24 (or30)p, sight unseen, over the Sony Z-1 @ 1080/30i any day of the week. The places where the HDV codecs break apart are quck movement, like unpredictable documentary moves on the pitching of a boat, and subtle gradients, like wide expanses of sky.

And while it's true that you can get miniDV tapes anywhere in a pinch, I don't think that's going to be an issue here. I also wouldn't just slap any old off-brand miniDV tapes into the Sony Z1 and expect them to be ultra-reliable, especially if you mix tape stocks. I've had a handful of brand-new DV tapes fail on me over the years, and I've only had one IDE hard drive go south (with warning enough to get the data off, BTW). I wouldn't get too dogmatic about tape being the safest and best storage soulution out there.

Let's say you have the ability to store your external drives in a nice, padded Pelican case, you treat them with reasonable care, keeping them away from undue moisture and shock, and you go the extra step to make a redundant backup on a second drive. In that case, I would say that you have an archival system that rivals, if not exceeds the reliability of MiniDV tape.

And, you get the DVCProHD codec, which will look better right off the camera, and hold up MUCH better to the inevitable post-processing. I've never been much of a Panasonic guy, but this camera is going to be unreal. And for everyone that says the workflow isn't there yet, I beg to differ. The workflow is laid out, right in front of you, and it's the same workflow we've all been using for years with our NLE's. You just have to start thinking about your footage as data, not as tapes.

Who knows, there may be a major gotcha that will surface once this camera is shipped. Maybe the Firestore won't even be available by the time you need to gear up, which would put a big crimp in your workflow. I just wouldn't commit to a particular camera until you have to. In the meantime, why not rent a Z-1 and go shoot some test footage? That will tell you pretty quick if the image is suitable for your purposes. That way, when the HVX-200 ships, you'll already have something to compare to.

I think everyone is way off on the archive solution here. Archiving a year's worth of footage onto HDD's that've been sitting in a boat is nuts. The footage should only be archived onto DLT--this is by far the most reliable solution. If you check out Panasonic's white paper on the HVX, DLT is their recommended archive solution. I'd say shoot with the Firestore (bring a couple backups) and archive to DLT.

Peter

Why is this a good thing Scott?

We are looking at both North American and European broadcasters so this is iindeed a factor. Wouldn't 1080i be another option to 24p? I'm just concerned that at times the movement within frame may be to frenetic for 24p.

This is my big concern. I got back from a circumnavigation 4 years back and that environment is extremely hostile to electronics...heck, I was beginning to corrode! But having said that, I still like what the HVX200 offers from a quality standpoint, provided that it delivers.

I would never consider selecting my shots at the shooting stage...I plan only to delete those which are unusable. You are certainly correct about backing up any electronically stored data which would certainly increase the cost.

Brilliant! Thanks David. This publication was very informative. Used to work for the Beeb many, many years ago and it was interesting to get their take on HD.

I've put out a call to DLT Solutions Inc. and am hoping that they will get back to me with an answer. I'll let you all know.

Thanks so much for all the help guys - a great forum.
Bruce

Because with 1/3" CCD's having native 1080 pixels, the result would be poor dynamic range and noise. Graeme Nattress has explained this much better than I ever could. In fact, if you want technical explanations, search for every post Graeme has ever made.

I was lucky enough to have met Graeme at NAB this year, and he's one of the most down to earth, self-effacing guys you could ever hope to meet. He also happens to be a codec/compression super genius. His plugins are pretty darn cool, too. Google him - you'll see. I would trust his opinion implicitly.

Here's one of the posts I was thinking of:
www.dvinfo.net/conf/showthread.php?t=48575

People said the same thing about the smaller Nikon D2X sensor before it came out. Despite what people thought, the camera is very clean. Technology has come along way in the last few years. I expect the HVX200 to have the same sensitivity and dynamic range as the DVX100a(or close to it).

John

Yeah, It's hard to think of a more damaging environment to subject a piece of electronic equipment to than one year of salt water spray. I'm sure that's not covered by any manufacturers warranty. I'd maybe worry about backing up my CAMERA, just as much as backing up my footage!

I think that you're most likely right about the HVX200 having a nice quality image. But I just can't see any easy way of keeping a years worth of material on hand, on any other medium than tape at this point. Hopefully by the time your shoot rolls around, someone will have solved this problem. Perhaps a portable blue-ray DVD recorder or some such thing?

I've only recorded short sailing trips and the King's Cup regatta in Phuket, but you do get some amazing footage. I envy your project!

Hi Bruce, interesting challenge you've got. Firstly DLT would probably be a good solution for your archiving needs. Obviously Hard Drives offer the best value, but DLT tapes are extremely robust. Generally you can drop them from several feet and they'll be fine. And that's without being in a plastic case. Maybe even consider wrapping them up in plastic wrap to really keep that moisture out.

Maybe I missread some of your post, but it sounds like you're going to be out on that boat for long durations. Being an IT person and having spent several years managing computer systems, I would like to mention redundency. Again, not sure about your exact scenario, but I doubt you'd want to be on the water with 2 months to go and have to sit around not shooting video because a cheap $100 hard drive broke down. To this end I'd recommend:
* A PC (or mac, whatever) with at least 2 hard drives configured in a hardware RAID.
* A spare hard drive that matches the ones in the computer.
* 2 External DLT drives. Yes, its expensive, but rarely they CAN fail early on. I've seen it, stuff happens. The good news is you keep the second one in the package and Ebay it off when you get home.
* A spare camcorder. Ok, you might not be able to cough up 6K for a second HVX, but maybe $1800 for a Sony HC1? I'm assuming some HDV footage would be better than no footage, should the HVX break down/drop in the ocean, etc.. Again, maybe even keep it in the box and sell it off when you're done.

Ok, I think I'll call it quits there. You still have some single points of failure there, but I think that would cover you pretty well. Remember, the cost of adding RAID and 2 drives will be pretty small and you'll recoup the majority of the money you spend on the extra DLT and camcorder.

Now if you're not going to be at sea for weeks or months at a time and can pick these things up easily, then of course most of this is overkill!

If budget simply doesn't permit the purchase of spare items, at the very least try to get a computer with a cheap on-board hardware RAID and the two drives. Its quite affordable and at least provides some protection against hard drive failure.

Good luck!

Philip Williams
www.philipwilliams.com

One of the best suggestions posts i've read- informative, toughtfull, and covering almost everything that can fail in the situation described by the original poster. This type of community sharing and help is what makes me coming here everyday.

i think philip gives really good advice, we did a job traveling through the panama canal and we backed everything up on dlt tapes and at every town we could we express mailed them back to nyc studio just in case the ship went down or our stuff was stolen...

How about another boat to put all this kit and editor on :)

Seriously I would consider HDV, possibly the Canon or JVC as it would simplify this particular workflow and in a years time Discovery and Nat Geo may not be so hostile to it as a format. The fact you're leaving in July will give you plenty of time to work this one out, but if it was me I would want the system with the fewest things to go wrong especially in the hostile conditions you may face.

Having said that, it would be worth trying a HDV camera on a boat first, some people have reported problems with HDV where every pixel in the frame is moving, this will certainly be happening on your trip, particularly in rough seas. I've shot with a Z1 on a boat to try and replicate this problem, although the sea wasn't too choppy I couldn't make the compression fail.

I forgot about that, the boat clearly represents a single point of failure :)

Ok, seriously, I did run across a post here or on dvxuser (I can't find the thread for the life of me now) where a guy posted direct frame grabs from an HD100. One of the frames had a fair amount of water and it definitely had serious MPEG2 blocking. I imagine the blocking would be mostly unnoticable during playback, but it might be bad enough for a picky Discovery Channel engineer to poo poo a project. If Bruce is thinking of going with HDV, I'd definitely recommend renting and testing to be 100% positive that it'll work under the circumstances of his project.

Just an FYI, I'm pretty neutral on this whole "HDV vs Other HD flavors" thing. I've liked all the HDV footage I downloaded. Given a choice I'd certainly prefer the HVX's discrete frames with 4:2:2 color, but I'll take decent HDV over standard DV any day. And there is something to be said for being able to record onto $5 tapes!

Philip Williams
www.philipwilliams.com

Wow! What a great response. Thank you all for the great support.
You're right. Very interesting stuff.
No kidding John! I managed to protect my DV camera from most of the elements on my last voyage and it's still going fine, but this time my HD camera (whatever it may be) will be protected with a raincoat in fine weather and an underwater housing in tough situations. I also have gimbled mounting points at various key areas on the boat which can be protected from the worst of the spray/weather. I wasn't able to get to Phuket on my last voyage - very sorry I missed it - but have a friend who raced in the King's Cup. Must have been great fun to film!
Yes Philip, I will be at sea/offshore for an extended period of a year or two, and thank you for the advice.
* I will certainly be having two laptops on board, but I question the need for a RAID array. If I'm going to be storing the material on DLT or on independent hard drives, what would the RAID protect? I will not be offlining on location so the computer will only be used to edit off unusable material and as an interface between the Firestore and the independent HDD.
* The computers will be matched.
* 2 DLT drives?...ouch! Okay...I'll check my budget. But it might end up cheaper to go the HDD method with matching HDD in a protected pelican as suggested by Scott earlier in this post.
* I've thought about the spare camera, but as I mentioned earlier, I was able to protect my DV camera quite well last time and I will be even more carefull this time. Also, there's no-where on this planet (well almost no-where) where you can't get stuff shipped into...and if it breaks...well the price might just have come down by then!
The Panama Canal trip was one of the highlights of my last voyage - hope you had a great shoot. Robert, could you tell me what DLT equipment you were using and if you have any suggestions for a workflow? The DLT company I contacted didn't feel their equipment was suited to the job.
Size IS important...and I reckon in this case the smaller the camera, the better so JVC is out. I also think that Canon might be on the heavy side, but I would be prepared to look at it. I'm in discussion with Discovery at the moment concerning format and other technical requirements, but you're right Steve, in a year or so when this project is ready for broadcast they might well have softened to HDV given the number of independents out there producing good stuff on HDV, but I'm not sure I can take that chance. The problem I'm worried about is exactly what you mention - the complex content within each frame. And btw, another boat??? Heck...I'd simply be happy with a BIGGER boat!!;)
You know...I think that this is really what it comes down to, and it always seems to be the same in this industry: Better quality=more money! Sigh....
Thanks again for everyone's input!
Cheers,
Bruce

Quantum VS160...

shoot... capture...backup...mail...we were working with dv files, we shot 8 hours of footage a day for 4.5 days...

we had dv tapes, footage on hardrive, and a backup on dlt (which was mailed)...we did some logging at night but after a long day somone else did it back in ny, so when we got back a rought draft was ready to go in a few days...bring extra laptop batteries, we only brought one laptop, two would be smarter, foul weather gear for your cam and all that good redundant stuff, lighting for shooting inside the cabin and at night...

i think the p2/hd workflow would be better then tape in this scenario, with tape it was such a pain to spend one hour per tape to back it up onto the hard drive, when sleep or other things were in order, 8 tapes a day took us 4 hours every night, our capture software was canopus that allowed us to capture up to 3 streams at a time, our lap top could only handle 2

backing up off a p2/hd leaves more time for logging and sleep...and a good nights sleep is everything

I would consider reliability to be the primary issue. Properly used, any of the 1/3" chip camcorders will give you good footage. They are all 1/3" cameras, and the HDV will be a little bit better than the DV, and the DVCPRO HD should be a little bit better than the HDV, but they're all still going to be fairly close because they're all 1/3" chip cameras. What I've seen of the Sony HDV from two different Z1 cameras looks pretty good, better than equivalent DV footage from a PD170. I would stick with tapes for proven, reliable sturdy technology. If I were going to do it, I probably would go with a Z1 and an FX1 for backup in case the Z1 got damaged. I'd also get an underwater housing--not the kind you use for diving with a camera, but one of those flexible ones good to about 10 feet, which should allow you to shoot in bad weather and with waves crashing over the bow.

Typically the most likely to fail component of a computer is the hard drive. A RAID array basically protects you from that failure. Obviously a hard drive failure in a single drive system = no computer. No computer = no working with video footage. For the little extra money that this adds to a PC (relative to 6K cameras and such), its well worth it for a critical application where replacement parts aren't readily available.

Now having said that, I guess its a mute point in your case since you'll be using laptops. If you're taking two laptops - and they're both capable of working with your HVX footage and offlining to external drives/tapes - then you're set. Forget redundent drives, you've got redundent computers! If you were traveling with one laptop I'd actually recommend removing the fully loaded hard drive (with OS, editing software, etc..) and then installing a second drive and loading the OS and software on that. Then if that drive fails you just drop it out and slip in the spare drive and boot up - instant work computer! A nice insurance policy for a couple bucks.

Philip Williams
www.philipwilliams.com

I would have thought HDV would have been *a whole lot* better than DV, but I guess the problem remains how the broadcasters view HDV in the context of HD...and I still don't have clarity from Discovery at this time.
You know Philip, it might be a damn good idea even with two computers. And as you so rightly said, it's not big money in the greater scheme of things.
Cheers,
Bruce

The BBC sees HDV as a consumer format (listed as such in the draft BBC Guide to HD referenced earlier in this thread). At the London Video Forum, held earlier this year, the BBC HD man speaking there said that HDV was not accepted as HD for co-production with partners like Discovery, and it would not command the HD premium payment rate. HDV sounds like something you will need to negotiate if this view still prevails at the Discovery channel.

HDV seems to be at the same point DV was just a few years ago. Eventually lots of people will be shooting HDV and making HD dubs for TV and for festivals, just as they now shoot DV and make Betacam and Digibeta dubs (and sometimes HD) for festivals, local TV, etc. No 1/3" chip HDV camera is going to replace a 2/3" chip HD camera, just as a 1/3" chip DV camera doesn't replace a 2/3" chip DV or DV50 camera. But these cameras are already significantly better than their DV predecessors.

Having owned numerous sailboats (the last one being a 34-footer) I can attest to the difficulties of electronics and a sailing environment. I would not want to be burdened with computers and hard drives when trying to shoot video, and I wouldn't want to always have to transfer my original footage to a drive and then back it up and then delete what I shot on so I could reuse it. To my way of thinking, that's a studio environment type of workflow. And I wouldn't want to do a gig like that without a second camera.