Harnessing the Firehose

Jon Fairhurst · June 13th, 2013, 03:44 PM

I'm interested to know what others are doing to manage the massive amount of data from RAW workflows, professionally. The goals are to 1) ensure that everything is stored at least twice, 2) the production is never slowed down, and 3) costs are minimized.

Specifically, we're thinking of shooting green screen RAW using the 5D2 and Magic Lantern. Apparently, we can shoot a 1728x972 crop continuously with just a fast card - today. The BMC Pocket cam would get us to 1920x1080 for only $1k, once available.

Sure, I'd love to shoot this on a three chipper, but that's out of budget, as shown here: http://www.dvinfo.net/forum/open-dv-...en-camera.html

Our project includes comedy improvisation, so we will often want to let the camera roll. That means we have to deal with a firehose.

On one extreme, you buy lots of camera cards and simply fill them. The Data Wrangler's job is to catalog the cards and make backups. Nothing gets erased. But that's expensive.

On the other extreme, one builds a giant array of SSDs with redundancy and copies a the camera cards lickity-split over USB-3.

One can use large 7,200 RPM drives for cost effective storage, but it's easy to hit a bottleneck and slow production.

We're thinking of a hybrid approach with a small number of camera cards, a small,redundant RAID of SSDs for offloading during the shoot, and offloading to large, 7,200 RPM HDDs overnight. This might be the most cost effective approach but it requires a lot of data handling and clearing of data, which creates risk.

There is also the possibility of coding to an intermediate format, such as Cineform RAW, to balance storage costs with quality, but transcoding can lead to yet another bottleneck.

The bottom line is that there's more to the cost of RAW than just the camera. I'd like to know what people are doing to make it all work. :)

Brian Drysdale · June 13th, 2013, 05:16 PM

Television productions tend to avoid RAW, especially if there's a tight schedule. Cameras like RED use compressed RAW, but I assume you're discussing uncompressed RAW. For ease of editing you're going to need a proxy and so far as I'm aware you can't currently edit the Cinema DNG format.

For what you're doing I wouldn't use RAW, I'd record to ProRes or DNxHD, which I believe is an option on the Black Magiic camera or other video codec.

Jon Fairhurst · June 13th, 2013, 05:46 PM

True, for the BM Pocket Camera, ProRes would be the sensible plan.

However, we have the 5D2 today and the Pocket is not yet shipping. All we would need are fast cards in order to shoot a 1:1 crop with 14-bit depth. That would give us good resolution (although Bayered), reasonable DoF with the crop sensor view, and a minimal camera cost. Color correction is very important to us as we will be matching the look to Hollywood scenes, so RAW offers tangible advantages.

And fortunately, we will be shooting in our "studio", so we don't have to worry about battery power, packing, and transport. The data wrangling issue is much tougher on-location or with multiple, moving setups.

Anyway *assuming* that we will shoot RAW, what do people recommend? We want to consider the cost, risk, and speed of the workflow before we choose or reject the format.

Chris Barcellos · June 14th, 2013, 03:52 AM

Jon

Conversion to Cineform works pretty clean so far for me. You might even consider. With the BMCC files I can process directly off the SSD by dragging the folders into GOPro Studio, No actual file transfer needed. Not getting it to that with my CF card from the Canon. Not sure what hangs it up, so I actually transfer to drive and convert. Convert itself seems really fast.

Tim Polster · June 14th, 2013, 03:32 PM

Jon, how much data/filming are we talking about here? When I hear "massive array of SSDs" I think of a budget issue somewhere. If you need to spend big in order to use a "cheap" camera, why not spend on a better camera so you do not need the support costs? Just a thought.

Renting a really nice camera with a better suited workflow might be a good option. But that depends upon your project. It is strange to have a mix of 14bit ultimate quality, greenscreen and no budget in the same sentence! :)

June 13th, 2013, 03:44 PM	#1
Jon Fairhurst Inner Circle Join Date: May 2006 Location: Camas, WA, USA Posts: 5,513	Harnessing the Firehose - RAW Data Wrangling I'm interested to know what others are doing to manage the massive amount of data from RAW workflows, professionally. The goals are to 1) ensure that everything is stored at least twice, 2) the production is never slowed down, and 3) costs are minimized. Specifically, we're thinking of shooting green screen RAW using the 5D2 and Magic Lantern. Apparently, we can shoot a 1728x972 crop continuously with just a fast card - today. The BMC Pocket cam would get us to 1920x1080 for only $1k, once available. Sure, I'd love to shoot this on a three chipper, but that's out of budget, as shown here: http://www.dvinfo.net/forum/open-dv-...en-camera.html Our project includes comedy improvisation, so we will often want to let the camera roll. That means we have to deal with a firehose. On one extreme, you buy lots of camera cards and simply fill them. The Data Wrangler's job is to catalog the cards and make backups. Nothing gets erased. But that's expensive. On the other extreme, one builds a giant array of SSDs with redundancy and copies a the camera cards lickity-split over USB-3. One can use large 7,200 RPM drives for cost effective storage, but it's easy to hit a bottleneck and slow production. We're thinking of a hybrid approach with a small number of camera cards, a small,redundant RAID of SSDs for offloading during the shoot, and offloading to large, 7,200 RPM HDDs overnight. This might be the most cost effective approach but it requires a lot of data handling and clearing of data, which creates risk. There is also the possibility of coding to an intermediate format, such as Cineform RAW, to balance storage costs with quality, but transcoding can lead to yet another bottleneck. The bottom line is that there's more to the cost of RAW than just the camera. I'd like to know what people are doing to make it all work. :) __________________ Jon Fairhurst

June 13th, 2013, 05:16 PM	#2
Brian Drysdale Inner Circle Join Date: Feb 2006 Location: Belfast, UK Posts: 6,143	Re: Harnessing the Firehose - RAW Data Wrangling Television productions tend to avoid RAW, especially if there's a tight schedule. Cameras like RED use compressed RAW, but I assume you're discussing uncompressed RAW. For ease of editing you're going to need a proxy and so far as I'm aware you can't currently edit the Cinema DNG format. For what you're doing I wouldn't use RAW, I'd record to ProRes or DNxHD, which I believe is an option on the Black Magiic camera or other video codec.

June 13th, 2013, 05:46 PM	#3
Jon Fairhurst Inner Circle Join Date: May 2006 Location: Camas, WA, USA Posts: 5,513	Re: Harnessing the Firehose - RAW Data Wrangling True, for the BM Pocket Camera, ProRes would be the sensible plan. However, we have the 5D2 today and the Pocket is not yet shipping. All we would need are fast cards in order to shoot a 1:1 crop with 14-bit depth. That would give us good resolution (although Bayered), reasonable DoF with the crop sensor view, and a minimal camera cost. Color correction is very important to us as we will be matching the look to Hollywood scenes, so RAW offers tangible advantages. And fortunately, we will be shooting in our "studio", so we don't have to worry about battery power, packing, and transport. The data wrangling issue is much tougher on-location or with multiple, moving setups. Anyway assuming that we will shoot RAW, what do people recommend? We want to consider the cost, risk, and speed of the workflow before we choose or reject the format. __________________ Jon Fairhurst

June 14th, 2013, 03:52 AM	#4
Chris Barcellos Inner Circle Join Date: Nov 2005 Location: Elk Grove CA Posts: 6,838	Re: Harnessing the Firehose - RAW Data Wrangling Jon Conversion to Cineform works pretty clean so far for me. You might even consider. With the BMCC files I can process directly off the SSD by dragging the folders into GOPro Studio, No actual file transfer needed. Not getting it to that with my CF card from the Canon. Not sure what hangs it up, so I actually transfer to drive and convert. Convert itself seems really fast. __________________ Chris J. Barcellos

June 14th, 2013, 03:32 PM	#5
Tim Polster Inner Circle Join Date: Dec 2004 Location: Arlington, TX Posts: 2,231	Re: Harnessing the Firehose - RAW Data Wrangling Jon, how much data/filming are we talking about here? When I hear "massive array of SSDs" I think of a budget issue somewhere. If you need to spend big in order to use a "cheap" camera, why not spend on a better camera so you do not need the support costs? Just a thought. Renting a really nice camera with a better suited workflow might be a good option. But that depends upon your project. It is strange to have a mix of 14bit ultimate quality, greenscreen and no budget in the same sentence! :)