SI-2K's RAW to DPX through Mac / Glue Tools
 

Hi David, I hope you can guide us a little with this. Please bear with me as I unravel my woes, as I haven't find a way for making this post terse.

We are currently editing a feature film which was shot using Silicon Imaging SI-2K cameras. Unfortunately our production company did not plan in advance for a coherent workflow, vendor-wise, and thus all the subsequent postproduction links in our chain are mismatched: for picture editing, instead of Adobe Premiere, we're using Final Cut Pro (Mac Pro Quad & Octo-core); and instead of IRIDAS SpeedGrade our production company owns a SCRATCH workstation, and is obviously intent on using it for this feature.

(—Oh boy. —Yes, I know, this is not going to be easy...)

Since SCRATCH works with DPX, we need to devise a way to bring the SI-2K original footage to DPX, as clean as possible, and retaining the maximum of its full RAW information. But besides the footage we also need to bring to SCRATCH the sequences cut in Final Cut. And since this is a feature film, we need to carry both, footage and sequences, in an efficient and speedy manner, so that conforming and grading can be done within a reasonable time frame.

The obvious choice for converting CineForm RAW footage to DPX is, of course, the cf2dpx.exe utility that comes with Neo and Prospect. With this utility we are able to convert the SI-2K/CineForm RAW .AVI footage to DPX on a file-by-file (shot-by-shot) basis. The resulting DPX files are both readable and addressable (TC-wise) by SCRATCH, and thus automatic conforming using an EDL that references files in the comments is possible. But this method becomes impractical from the point of view of a feature film, where more than 1500 individual files would have to be converted manually using the command line, a devastatingly time-consuming procedure prone to human error. And cf2dpx.exe is a PC-only utility, so we can't run it in our Macs.

Now, our CineForm RAW footage is being edited within the Mac/FCP/QuickTime environment, albeit we are using proxy footage (ProRes 720p) for the actual editorial work, instead of the original CF RAW files. We resorted to proxies because when we started the project we knew little about CineForm and Mac, and we did not know that CF RAW footage edited in a ProRes sequence allowed for real-time effects; since we had tons of effects coming, working with the original media was ruled out in favor of ProRes proxies. Reconnecting our FCP sequences to the original .AVI CineForm RAW footage is trivial, since we kept the same names for both sets of files (CF RAW and Pro Res). This means that at any time we can make a FCP sequence reference the original 2K RAW footage in a snap.

It followed that if we could export such a sequence into DPX we could, in theory, bring a preconformed result to SCRATCH, which would only need to be "notched" using an ordinary EDL for subsequent grading. Enter Glue Tools, a third-party QuickTime component which allows for DPX and Cineon import and export from Macs, and specifically, from Final Cut Pro and Compressor. Our first tests showed that such an export to DPX would go nicely into SCRATCH. But we did not know, and do not know now, which parameters to use in the conversion, and how to use the Tools, and so here, at last, come our questions.

First, our RAW footage was delivered as .AVI files because Silicon Imaging recommended to our camera department NOT to use the .MOV (QuickTime) wrapper option, as it was not fully tested. Now, I've browsed through CineForm's site, and I've just discovered that .AVI files are processed as 8-bit within the QuickTime/Mac environment. If we are planning to export DPX files using the Glue Tools components, does that mean that we need to convert all the incumbent footage to .MOV in order to preserve the 10- or 12-bit range of the RAW files in the export files? If so, is there an automated way or batch process for this .AVI to .MOV conversion?

Glue Tools offers three different conversion modes to (and from) DPX/Cineon: linear, log-lin, and RAW. We haven't been able to figure out which is the "right" mode for our case, and Glue Tools developer, Bob Monaghan, keeps telling us that he cannot offer any guidelines on this respect, nor he would recommend which parameter values (neg gamma, display gamma, white point, black point) to use, instead urging us to contact our lab for those "magic" values. Since our production company is, in this respect, our own "lab", we are left to our own devices for devising which setting and values to use to convert from CineForm RAW to DPX. Of course, we know there is no such thing as "magic" values and that every case has its own characteristics, but we need to start at least with a ballpark estimate. How should we convert CineForm RAW to DPX using something like Glue Tools? What's the theory behind this conversion? Should the resulting DPX be log or linear? Should we mingle with the logarithmic parameters (neg gamma, disp gamma, white point, black point) or leave them at certain "standard" values for CineForm RAW?

There is a certain possibility that we are doing more than one thing wrong here, or making the wrong assumptions. Any help would be very much appreciated. Thank you for reaching this far in the post, and I beg you forgive my lack of brevity.

Best regards,

Alex
Editor, "La Leyenda"
Pampa Films S.A.
Buenos Aires
Argentina

Alejandro, unfortunately we have never used Glue Tools, so we don't have any precise guidance for you about this.

I believe Jason Rodriguez at SI has used Glue Tools in a manner that you're asking about, so he may have more specific guidance.

Regarding AVI versus MOV - yes, the AVI importer used by Apple will only support 8-bit depth. You can use CineForm's HD Link software to rewrap the AVI files to MOV on Windows, then move the MOV files back to Mac and they will be properly interpreted at high bit depth. Check resulting timecode on these files to make sure the workflow remains consistent.

Regarding which curve to apply when converting to DPX, the comment from Glue Tools is correct - we can't tell you which curve to use. But generally you want a curve (somtimes log) applied, you don't normally want a linear gamma. As these files will be used in Scratch you might want to check with Assimilate to see if they have recommendations based on the end purpose that you intend.

David, many thanks for such a prompt answer.
Would you mind sharing his email address, so I can reach him and delve into the specifics of our situation? My own email is alexkrilow@yahoo.com.
I wonder, does it mean that this conversion process is known to alter the timecode of the resulting files, or your suggestion is just a customary double-check procedure?
Please forgive my ignorance, but when you say "apply a curve", do you mean those pesky parameters, which in GlueTools are called "negative gamma", "display gamma", "white point" and "black point"? Do those four parameters represent or define "a curve" that is applied to the CineForm RAW original files?

I just want to understand what kind of operation is taking place whenever I translate CineForm RAW to DPX. In this respect I have a more general question: what is the "nature" (log/lin) and bit depth of CineForm RAW files? I have been told that they are "linear 12-bit files, much like a scanner's sensor data". I think there is more than that to this story.

Again, thanks a lot.

PM sent....

Here is are suggestions for working with the Glue-Tools plug-in along with FCP

1) Media Manage your finished "picture-locked" timeline. This will gather all the media that is contained in that sequence into one single folder, along with a Final Cut Pro project that just contains the media and the single master sequence.

2) Take the AVI files from the media managed project folder and re-wrap them to QT using the HDLink utility from CineForm. Although this utility runs on Windows, it can do batch processing by simply pointing it towards a folder.

3) Once you have moved all your media-managed AVI's over to QuickTime, then use compressor, and with Glue-Tools installed, you will see that there is an option in compressor to export a DPX sequence from a movie file.

4) Setup GlueTools using the "RAW" mode, or with the "Linear" mode using the black point set to 0 and the white-point set to 1023. I would suggest though the RAW mode for best results at this point that avoid gamma problems. Now one thing to note is that inside of Compressor you will probably see a slight gamma shift, but that is simply a bug with the QuickTime decoding of DPX files. If you compare the output DPX to the AVI file in something that does not use the QT engine, such as After Effects, you will see that the gamma is correct between the two files.

5) Process all your clips using Compressor (not FCP), and export them out into a new series of folders. Make sure the naming template you use in Compressor will be something that Scratch will be able to conform to using the names in the EDL

6) Go back to FCP, and using the master timeline, export and EDL file. I would use the "clip names" option rather than the "file names" option so that Scratch does not get confused by file extensions.

7) Take the EDL and import it into Scratch, and then point the EDL to the DPX file sequences you exported using Compressor.

Some points for the future:

1) You can do a cut using the AVI's in FCP and conform them in SpeedGrade XR and completely avoid the entire conversion process described above. SpeedGrade will read the AVI's in their native file format, and there won't be any 8-bit truncation. SpeedGrade is agnostic in that respect when it comes to AVI vs. MOV. The 8-bit limitation issue is purely with FCP and the QuickTime engine itself.

2) You could write a batch script for the cf2dpx.exe utility to automate functionality and reduce human error. I would run this under boot-camp for the MacPro's, or even Parallels (so you don't have to reboot).

3) You can now record QT's in the camera, although again, QT uses up more processing power, so you will have to use the lower-quality compression modes around 8-10:1 rather than 3.5:1 and 5:1. That being said, I would evaluate the 8-10:1 modes before you get concerned about the picture-quality, as they can look very good as well.

Thanks,

Jason

One more thing:

Before you do the actual DPX conversion, make sure to select White-Balance + Matrix in the "Set Active Metadata" applet that comes with Neo4K on the Mac. This will give you the correct color on your clips so that you can color-correct them properly. Not selecting this option will give you un-color-calibrated "RAW" images, that if you do not have the proper tools, could become hard to get the correct color-information out of (a lot of color-correction tools don't have color matrix controls to properly set the color-space of the camera).

Thanks,

Jason

Jason, your reply is exactly the information I was looking for, and it could not have been any clearer or or more detailed. On behalf of all our postproduction team let me express our deepest gratitude to you.

I still have a couple of questions. If I understood your explanation correctly, we could go to DPX using either HDLink + Glue Tools + Compressor, or cf2dpx.exe within a script for batch processing. Could you tell us what the differences are between these two paths, besides the obvious operational issues? Is the result of one translation better than the other in terms of visual quality and/or information? What would you suggest?

And following your explanation, I assume that in any case I will need to bring entire takes to the SCRATCH workstation, and conform the reels within SCRATCH. I used to have this notion that using Glue Tools I could export preconformed DPX reels (i.e., sequences carrying only the "used media") from Final Cut (using Compressor, mind you), then import them into SCRATCH, and just use an EDL to "notch" them (v.g. add edits so that each shot is delimited for grading). Of course I could only do this using the Glue Tools method, not with cf2dpx.exe. Would you advise against this? If so, why?

Again, Jason, thank you very, very much for your time and information. And also our thanks to David for forwarding our inquiry to you. We really needed this. Best regards,

Alex

The main advantage to using the CF2DPX approach is that there is no re-wrapping required . . . you can do the DPX conversion directly on the AVI files.

The visual quality between HDLink+GlueTools+Compressor vs. AVI->DPX using CF2DPX will be the same.

For right now though, you will have to render all your takes, there is no methodology for just rendering the DPX files needed from a conformed EDL. This is because right now the media manager inside of FCP will not trim CineForm clips, it simply copies the entire clip into the media-managed directory. So if you use any portion of a clip on the timeline, you will have to convert the entire clip to DPX.

Got it; that's one thing I wanted to know about FCP's media manager handling CineForm RAW .AVIs: no "consolidation" is possible. It makes sense.

What about exporting an already conformed DPX sequence from Final Cut Pro, directly from a FCP timeline, but via Compressor? This is like when you want to export a QuickTime movie from a Final Cut Pro sequence or from the timeline, but using the option "Export using Compressor..." from FCP's menu. This "sends" the sequence ("the timeline") to Compressor, and the actual conversion to DPX is handled by Compressor / Glue Tools —and not by FCP. If I am not mistaken, this would guarantee that bit-depth gets preserved in the conversion. The result in this case is a single GlueTools-processed DPX sequence, and not a media-managed folder with original CF RAW .AVI files to which an EDL is referring.

If I could do that, instead of transferring entire files ("takes") out from FCP, I could only transfer, say, a single DPX sequence for each conformed reel. That would be very efficient in terms of time for us. Is this procedure possible? Is it safe?

Thanks, as ever.

Yes, this will work just fine. The only thing to keep in mind at this point is that you won't have individual cuts. But I'm guessing that's okay.

Thanks,

Jason

BTW, make sure that you do the actual render of the timeline with Compressor, not with FCP itself . . . that is very important to maintain the 32-bit floating point pipeline, and not lose bits or have your processing pipeline truncated to 8-bits accidentally.

Thanks,

Jason

Many thanks, Jason. We will make some tests using all the information you have provided. Hopefully you have set us on the right track, but if we run into trouble I will post again.

Best regards,

Alex