Restore clipped highlight detail in Premiere Pro CS3 with superwhites
 

The HDV files created by my Canon XH-A1 HDV contain a lot of detail in the superwhites: the 235-255 or 100-110 IRE range. Premiere Pro normally clips that detail, but I found a way to get at it: add the Fast Color Corrector effect and set the output level to 215. This restores the highlight detail in the program window as well as all the output options, including still frame extraction, which has the superwhites mapped to 255.

Unfortunately, adding any other video effect clips the highlights again, even if the Fast Color Corrector is isolated from the other video effects by nesting a sequence.

I would have expected 235 to be a sufficient output level to restore the superwhites, but it actually took exactly 215, not a hair more or less. Sony Vegas shows the IRE clipping at 110 in its YC Waveform, but Premiere doesn't clip until around 117 or so on the same frame.

I tried using Sony Vegas, and it impressed me greatly by saving the superwhites automatically: no extra work necessary. Furthermore, it doesn't clip them in any of the video effects, including curves and levels.

I also tried converting the files to Cineform (Neo HDV trial), but the highlight detail was still clipped when viewed with avisynth+virtualdub.

What does all this mean? Is there any other or better way to restore superwhites in Premiere?

Interesting post!

Just to clarify ... when you say "I also tried converting the files to Cineform (Neo HDV trial), but the highlight detail was still clipped when viewed with avisynth+virtualdub", are you meaning when you converted the raw files, or when you exported your "FCC plus 215 output" files?

The raw files that I captured with hdvsplit.

"Premiere Pro normally clips that detail"

I don't think that it's a function of Premiere that does the clipping..Simplified, the legal video signals are being clipped..Whether Vegas or FCP.

Remember, you can theoretically go above 100%, since Chroma allows for more padding (can't remember exactly how much)....

Nonetheless, if you were truly clipping to tape... the import, or editing for that matter couldn't possibly bring back information that's been blown away...
I believe that you're close, but haven't clipped.

As far as bringing down your highlights, i'd personally attack the tonal range first..
Drop the Master Higlights, bring down the Lift, and play with the gammas to taste.
IIRC the RGB color corrector had these options in the slider...

I would only use fast color corrector for reference...

Peter, for me, the RGB Corrector clips everything over 100 IRE. Are you saying that you can successfully bring 110 IRE (255 RGB) down to 100 IRE in HDV files?

Hey Daniel....

Firstly, let me get back to you on the correct filter. I'm running on memory currently..
Secondly, you can bring 110 IRE down to 20 IRE if you desire. My point being, unless you've applied filters, Premiere's RGB parade (or external monitor) should show you the same as your source footage. Nothing more, nothing less. And if you've blown the highlights at tape, then nothing can be done to bring that information back..Which leads me to believe you haven't blown them yet....

Be careful about your method..Depending on the feel and source, you're remapping all pixels, including blacks...That might not be desired..In your case though, it might be acceptable..
By playing with Higlights and Gammas, you shouldn't be affecting your shadows too much...

I'll get back to you on the filter...

In the RGB parade there is a lot of data that appears to butt up against 100. The program monitor shows clipped highlights (without any filters). If I add the FCC 215 output level, the RGB parade drops to just below 90 and the highlights appear again the program monitor.

Good point. In this case I'm planning to crush a wide range of shadows with an RGB curve, so I'm flexible on that.

Thanks.

Yeah, i was right, it was the RGB color corrector...

I've sampled some DV footage. You might find the gain to be very sensitive..
But at least, you don't effect the Pedestal or Gamma drastically..
Regardless, my clipping won't bring back lost detail....No matter how far down i bring it...It was blown out at the tape..

It's always a challenge taping African men (Burundi) against a brightly lit window :).


Daniel wrote:
"the RGB parade drops to just below 90 and the highlights appear again the program monitor."

Then as expected, they weren't totally blown out to begin with. So that's good news.

Test Video with detail in the superwhites.
 

I found that RGB Curves can bring down the superwhites easily, but I'm still unable to do it with the RGB Color Corrector, Adjust->Levels, or any other filter.

I shot a quick test video. I would appreciate it if a few people here could throw it on a Premiere project and see how the superwhites are handled on their system.

For me, here's what it looks like with no effects (clipped superwhites):

http://thebrownings.name/video/test/...no-effects.jpg

And here's what it looks like after bringing the superwhites down with FCC or RGB curves.
http://thebrownings.name/video/test/...te-FCC-215.jpg

This is the file captured off tape with hdvsplit. It's XH-A1 HDV 1080p24F.

http://thebrownings.name/video/test/...t-XHA1-24F.m2t

Daniel, the image looks the same here, with and without Fast Color Corrector applied.

However....I can also get much the same result with Adjust/Levels (by reducing RGB White) and with RBG Color Corrector (by reducing gain).

So when you say "you can't do this" with other filters, what are you meaning exactly? That the suoerwhites disappear? Is so, thats not the case with my computer.

Thank you for checking, Graham. So when you load the m2t in Premiere, you see the pattern detail (dots and diagonal lines) all over the paper towel? Without applying any effects?

For me, the paper towel looks solid white in the Program Monitor, like the first frame I posted (superwhite-no-effects.jpg). I can only get the pattern detail of the paper towel to appear if I apply the Fast Color Corrector or RGB Curves, in which case it looks like the second frame I posted (superwhite-FCC-215.jpg), which is the look that I desire.

Yes, I mean they disappear in the sense that instead of showing what the paper towel looks like, it just becomes solid white.

Your post has me thinking that something is broken on my installation. Time to put on the debugging hat. :-)

Sorry, I was unclear in that last email.

What I meant was .... I see the same thing as you with the two images - i.e. no visible detail in the timeline window with the raw footage; detail appearing in the paper towel once fast color corrector is applied and the output level is reduced.

BUT .... what I'm also saying is that I can recover that same detail with RGB color corrector (with gain reduced to 0.9) or with levels (with RGB white output reduced to 220).

I see now, thanks Graham. I've tried every which-way combination I can think of, but I still can't make Levels or RGBCC work for me. If anyone has the opportunity to try out this little m2t file, I would appreciate it.

You say you can't recover that detail with RGBCC ... so I'm wondering what happens for you when you reduce gain?

Are you saying that the towel shifts from uniform white to uniform grey, so that none of the texture detail ever appears?

Yes. I add the RGBCC effect to the clip, change nothing except gain to 0.90, and it does exactly as you say: the towel shifts from uniform white to uniform grey; no detail ever appears.

The same thing happens if I use Adjust->Levels and set just the output RGB white to 215. (What's curious is that the same levels adjustment done via FCC *does* work.)

I've also tried creating new project files with different settings. Tried changing bit depth preference, interpret footage, and a few other things. I tried changing a bunch of options in the RGBCC: Tonal Range, Tonal Range Definition, adjust each channel's gain individually, etc. All to no avail.

Wow. That IS weird!

Sorry for the late response but I only just ran into this thread. I tested the provided footage and my Premiere CS3 behaved exactly like Daniel Browning described. I opened a fresh project with the basic NTSC HDV settings, and both the Levels and RGB Color Correction filter clips the whites, while Fast Color Correction does not. The scopes show that just adding either of those two bad behaving filters to a piece of footage, results in the whites being clipped, even if the filter settings were not touched. Lowering the white levels does not bring back the details, they stay clipped.

Adobe Media Encoder forces clipping
 

I had a more thorough look at the can - here are the worms.

The following filters are good for bringing down the superwhites recorded by the camera: Fast Color corrector, Three-Way Color Corrector, Luma Curve, RGB Curves, ProcAmp.

Combining any of the following to the stack will clip the superwhites even if the above "good" filters have been used to bring them down below 100IRE: Levels, Shadow/Highlight, Color Balance, Color Balance (HLS), Tint, Gamma Correction, and probably most of the rest. Also, RGB Color Corrector and Luma Corrector cannot bring superwhites down and clip the superwhites if used before one of the "good" highlight adjusting filters. RGB Color Correction always clips the chroma.

All of this works when exporting to tape, or exporting to movie. Does not work when using Adobe Media Encoder, unfortunately!

Exporting via Media Encoder always results in the original superwhites being clipped, even if the superwhites were brought down to legal values with the "good" filters. It appears that Media Encoder first clips superwhites and only then applies the filters. MainConsept MPEG Video codec, or its implementation, seems to be buggy.

As a work around, before using Media Encoder, export the parts with corrected superwhites to movie, preferably with a non-lossy codec, and import back to your project.

Note that RGB Parade seems to always show whites as clipped, even if they are not. Use All Scopes or Vect/YC Wave/RGB Parade to check the RGB values. The other scopes work as expected.

What - there's a manual?
 

A minor update to my previous post:

Selecting the Maximum Bit Depth video rendering option in Project Settings makes the RGB Parade scope show the superwhites properly. Does not fix anything else, though.

The filters understanding superwhites are tagged as "high bit-depth" in the Adobe Premiere Pro Help manual.

Hey this is great info - appreciate the effort you have put into clarifying this!

Superwhite handling in CS4
 

I don't see anything in today's Premiere Pro CS4 announcement that makes me think this issue is fixed. (If it were, I would have expected Adobe to tout it's increased dynamic range handling and highlight recovery options.) But we'll see about it when it comes out.

I tested 4.0.0 and found no improvement. I continue to restore superwhites with RGB curves or FCC.

An update to an old post.

I'm using PPro CS 4.2.1. I haven't yet found a way carry detail from the superwhites in my DV source material to my rendered output.

I'm monitoring on a PAL CRT video monitor with an analogue external waveform monitor all fed from a Blackmagic Design Decklink Extreme card.

When I play my DV files from my timeline, the superwhites carry through to my video monitor and external 'scope but the superwhites are clearly clipping on the computer monitor's display, and likewise they always clip when I output using Adobe Media Encoder.

As soon as I add ANY filter (including Fast Colour Corrector or RGB curves) the superwhites clip on the external monitor. No amount of tinkering with Fast Colour Corrector or RGB Curves recovers the detail that is in the superwhites in my DV source material.

I may try AVIsynth later today... I have tens of hours of footage so exporting to a lossless codec will take ages to render and will consume a huge amount of disk space.

Thanks for the update. I wish I had some help for you. I did a quick test of Premiere Pro 5 and found that the superwhites are clipped on the computer monitor, but kept in the encoder (without any effects added). At least it's some improvement, but it's still not as good as Vegas (which displays the superwhites automatically).

Ah, well, if I can't get a solution to work with CS4 then perhaps this'll be a good excuse to update to CS5! I mostly care about the superwhites making their way into the encoded output; I can just about live with them clipping in the timeline previews

I just tested the latest patch (5.0.3) to Premiere Pro CS5 and found that it still needlessly tosses all the highlights and shadows in the garbage for editing preview and most export formats. How are other people dealing with this? Do you purposefully buy cameras that always limit themselves to 16-235? Or do you apply effects to each and *every* cilp to reverse the damage Premiere does to all of them? Do you just let the data go to pot and figure that you didn't really need that much dynamic range any way?

I find it incredible that when faced with the crippling limitations that 8-bit video imposes on us, Adobe found a way to make it even worse by throwing away some of what little data we have. Where is the outrage?

> Where is the outrage?

Part of my outrage became redirected to Premiere's inability to properly decode AVCHD footage: http://www.dvinfo.net/forum/adobe-cr...hroma-bug.html

Ewww... that is bad. Looks even worse than just a bad job of decoding 4:2:0.