Harnessing the Firehose - RAW Data Wrangling at DVinfo.net

Go Back   DV Info Net > The Tools of DV and HD Production > Open DV Discussion

Open DV Discussion
For topics which don't fit into any of the other categories.


Reply
 
Thread Tools Search this Thread
Old June 13th, 2013, 04:44 PM   #1
Inner Circle
 
Join Date: May 2006
Location: Camas, WA, USA
Posts: 5,513
Harnessing the Firehose - RAW Data Wrangling

I'm interested to know what others are doing to manage the massive amount of data from RAW workflows, professionally. The goals are to 1) ensure that everything is stored at least twice, 2) the production is never slowed down, and 3) costs are minimized.

Specifically, we're thinking of shooting green screen RAW using the 5D2 and Magic Lantern. Apparently, we can shoot a 1728x972 crop continuously with just a fast card - today. The BMC Pocket cam would get us to 1920x1080 for only $1k, once available.

Sure, I'd love to shoot this on a three chipper, but that's out of budget, as shown here: Best inexpensive green screen camera

Our project includes comedy improvisation, so we will often want to let the camera roll. That means we have to deal with a firehose.

On one extreme, you buy lots of camera cards and simply fill them. The Data Wrangler's job is to catalog the cards and make backups. Nothing gets erased. But that's expensive.

On the other extreme, one builds a giant array of SSDs with redundancy and copies a the camera cards lickity-split over USB-3.

One can use large 7,200 RPM drives for cost effective storage, but it's easy to hit a bottleneck and slow production.

We're thinking of a hybrid approach with a small number of camera cards, a small,redundant RAID of SSDs for offloading during the shoot, and offloading to large, 7,200 RPM HDDs overnight. This might be the most cost effective approach but it requires a lot of data handling and clearing of data, which creates risk.

There is also the possibility of coding to an intermediate format, such as Cineform RAW, to balance storage costs with quality, but transcoding can lead to yet another bottleneck.

The bottom line is that there's more to the cost of RAW than just the camera. I'd like to know what people are doing to make it all work. :)
__________________
Jon Fairhurst
Jon Fairhurst is offline   Reply With Quote
Old June 13th, 2013, 06:16 PM   #2
Inner Circle
 
Join Date: Feb 2006
Location: Belfast, UK
Posts: 4,121
Re: Harnessing the Firehose - RAW Data Wrangling

Television productions tend to avoid RAW, especially if there's a tight schedule. Cameras like RED use compressed RAW, but I assume you're discussing uncompressed RAW. For ease of editing you're going to need a proxy and so far as I'm aware you can't currently edit the Cinema DNG format.

For what you're doing I wouldn't use RAW, I'd record to ProRes or DNxHD, which I believe is an option on the Black Magiic camera or other video codec.
Brian Drysdale is offline   Reply With Quote
Old June 13th, 2013, 06:46 PM   #3
Inner Circle
 
Join Date: May 2006
Location: Camas, WA, USA
Posts: 5,513
Re: Harnessing the Firehose - RAW Data Wrangling

True, for the BM Pocket Camera, ProRes would be the sensible plan.

However, we have the 5D2 today and the Pocket is not yet shipping. All we would need are fast cards in order to shoot a 1:1 crop with 14-bit depth. That would give us good resolution (although Bayered), reasonable DoF with the crop sensor view, and a minimal camera cost. Color correction is very important to us as we will be matching the look to Hollywood scenes, so RAW offers tangible advantages.

And fortunately, we will be shooting in our "studio", so we don't have to worry about battery power, packing, and transport. The data wrangling issue is much tougher on-location or with multiple, moving setups.

Anyway *assuming* that we will shoot RAW, what do people recommend? We want to consider the cost, risk, and speed of the workflow before we choose or reject the format.
__________________
Jon Fairhurst
Jon Fairhurst is offline   Reply With Quote
Old June 14th, 2013, 04:52 AM   #4
Inner Circle
 
Join Date: Nov 2005
Location: Elk Grove CA
Posts: 6,838
Re: Harnessing the Firehose - RAW Data Wrangling

Jon

Conversion to Cineform works pretty clean so far for me. You might even consider. With the BMCC files I can process directly off the SSD by dragging the folders into GOPro Studio, No actual file transfer needed. Not getting it to that with my CF card from the Canon. Not sure what hangs it up, so I actually transfer to drive and convert. Convert itself seems really fast.
__________________
Chris J. Barcellos
Chris Barcellos is offline   Reply With Quote
Old June 14th, 2013, 04:32 PM   #5
Inner Circle
 
Join Date: Dec 2004
Location: Arlington, TX
Posts: 2,230
Re: Harnessing the Firehose - RAW Data Wrangling

Jon, how much data/filming are we talking about here? When I hear "massive array of SSDs" I think of a budget issue somewhere. If you need to spend big in order to use a "cheap" camera, why not spend on a better camera so you do not need the support costs? Just a thought.

Renting a really nice camera with a better suited workflow might be a good option. But that depends upon your project. It is strange to have a mix of 14bit ultimate quality, greenscreen and no budget in the same sentence! :)
Tim Polster is offline   Reply
Reply

DV Info Net refers all where-to-buy and where-to-rent questions exclusively to these trusted full line dealers and rental houses...

Professional Video
(800) 833-4801
Portland, OR

B&H Photo Video
(866) 521-7381
New York, NY

Z.G.C.
(973) 335-4460
Mountain Lakes, NJ

Abel Cine Tech
(888) 700-4416
N.Y. NY & L.A. CA

Precision Camera
(800) 677-1023
Austin, TX

DV Info Net also encourages you to support local businesses and buy from an authorized dealer in your neighborhood.
  You are here: DV Info Net > The Tools of DV and HD Production > Open DV Discussion

Thread Tools Search this Thread
Search this Thread:

Advanced Search

 



Google
 

All times are GMT -6. The time now is 04:59 PM.


DV Info Net -- Real Names, Real People, Real Info!
1998-2017 The Digital Video Information Network