The Great Camera Shootout Part 3

[Brian Drysdale]

EMMY nominated camera tests is a new one, but part 3 covers rolling shutter and skin tones.

The Great Camera Shootout 2011: SCCE ~ Episode Three | Zacuto USA

[Jon Fairhurst]

The next test - shooting the same challenging scene with various cameras - sounds interesting. They won't reveal the cameras until the end.

Regarding rolling shutter, I wish they would have measured it numerically with a strobe test and then shown a real world comparison. The spinning contraptions weren't numeric or real.

[Daniel Browning]

I love these. They're very entertaining, informative, and fun. I wish they tried a little harder on scientific method and factual accuracy, but hey, it's free.

[Brian Drysdale]

Is there a numerical standard for rolling shutter? I suspect how important artifact is depends on what you're shooting and you make a decision on that basis. So, if you're shooting a fast crabbing shot through railings, the DSLRs aren't the best camera to use, unless you wish to use the skew as a visual effect.

The spinning device is real if you're filming an aircraft propeller, but also does test things to destruction and so you do get a sense of the resistance to the effect that each camera has.

[David Heath]

I note they say "Some updates to certain cameras such as “S Log” on the Sony F3 and “CineStyle” for the Canon cameras were not available at the time of these tests.", then later

"After the screening, Raine went on to emphasize that the real difference between the cameras in this comparison was how much information they record and whether or not the cheaper cameras could hold up to substantial color correction in post."
Surely, for such as the F3, whether or not S-log is used or not will make a MASSIVE difference to such as post colour correction. Doesn't that make the tests as they stand a bit of a waste of time?

[Steve Nunez]

No GH2, sad!

[Brian Drysdale]

Originally Posted by David Heath

After the screening, Raine went on to emphasize that the real difference between the cameras in this comparison was how much information they record and whether or not the cheaper cameras could hold up to substantial color correction in post.

I suspect that would add to the complexity of a mass test, but it is important if you're going to do much heavy duty grading in post. DP Nick Paton did a test that might be something to consider as a possibility when he compared 7D and the Alexa.

Alexa vs 7d latitude tests on Vimeo

Of course, this is the last of three Zacuto "shoot out" videos, so you'd need to make any judgements a camera on all of these, plus other sources and your own tests.

[David Heath]

Originally Posted by Brian Drysdale

I suspect that would add to the complexity of a mass test, but it is important if you're going to do much heavy duty grading in post.

Yes, it will create practical difficulties (not least that I believe the option didn't come out until later!) but whilst S-log may not make much difference to some aspects of performance, to things such as post grading the use of native codec versus S-log is likely to make a vast difference.

In spite of the practicalities, then what is the use of any test that may say the differences between two cameras are small (the limiting factor is recording format, not camera head) - but when used in S-log mode one of them becomes vastly better?

[Jonathan Shaw]

One of the main things that really struck me was the rolling shutter on RED.... whay worse than I thought it would have been.

[Brian Drysdale]

Originally Posted by David Heath

In spite of the practicalities, then what is the use of any test that may say the differences between two cameras are small (the limiting factor is recording format, not camera head) - but when used in S-log mode one of them becomes vastly better?

Yes, in a comprehensive testing this should be covered, although there are a lot of apples and oranges being compared in these particular camera tests anyway. By next year, if Canon or RED have brought out a S35 camera that is in the F3 budget range, this would be an increasingly important deciding factor.

The differences between the RAW version and other codec versions of the same cameras could another factor.

With the number increasing numbers of specialist "motion" large sensor cameras coming out next year, I'm not sure how the DSLRs will quite fit into another shoot out, especially if there is no real improvement in these cameras or new models.

[Mikko Topponen]

Pretty stupid that in the rolling shutter test, all the DSLR's were zoomed in closer. Offcourse they will have more rolling shutter when closer to the object because then the object moves faster. Daa.

The bad rolling shutter on RED didn't really come as a surprise, there are many movies and music videos that are shot on RED that have quite a lot of rolling shutter. F3 has about the same as FS100/AF101.

1d mark IV has it pretty well under control, I guess Vincent Laforet wasn't lying when he said it did have significantly less rolling shutter than the 5d.

Still, I kinda wonder why in such a controlled test, the rolling shutter artifacts were tested with completely different FOV's on the DSLRs than on the regular cams. The results are not completely comparable.

[Brian Drysdale]

Only the last two were significantly tighter than the earlier cameras, but it's a temporal issue rather an image size artifact. The rolling shutter artifact shown on the disc is far larger compared to the other cameras than the slightly increased image size of the disc on the last two DSLSs. The disc is rotating in a flat plane more or less parallel to the sensor plane, so this image size difference wouldn't significantly affect how the spokes are picked up by the sensor.

[Jon Fairhurst]

Originally Posted by Brian Drysdale

Is there a numerical standard for rolling shutter?

You can measure the delay from the top of the picture to the bottom of the picture. For the 5D2, it is roughly 25ms. 24fps is 41.7ms per frame. At 24fps, that's 60% of the frame time. At 30p, it's 75% of the frame time. Having those numbers for each camera would let us compare them.

The way I tested the 5D2 is with a manual flash unit set to the fastest available speed. Film it. Find a frame that captured the flash from start to finish. Find a second set of frames where the flash started at the bottom of one frame and ended at the top of the following frame. Count the lines on each frame. From this information, one can calculate the rolling shutter latency as well as the actual shutter speed. Cool huh?

If I find a camera that has 10% less rolling shutter than the 5D2 (22ms), I don't really care. It's not that significant a difference. There will still be lots of jello. If it has half the rolling shutter (12ms), I'll be impressed. I will still want to control camera motion, but I'll know that the artifact will be significantly less noticeable. If the rolling shutter is 10% of that of the 5D2 (2ms), I'll be confident that it's virtually gone and I can shoot handheld on a boat in a storm.

[Brian Drysdale]

I guess it can be debated over giving numbers or just showing a visual test. There'll be people asking what the numbers look like if you just give a number, while others just like numbers.

[Jon Fairhurst]

I agree that we want both the numbers and real-world pictures. Unfortunately, the shootout test fell into the trough in between - it was geeky without being numeric or real.

For real images, I would have opened up the ASC handbook and designed motor controlled pan and tilt tests that fall just within the ASC limits. I believe that for 24p, you want 7 seconds for an edge to edge pan. For 16x9, that is the same speed as roughly 4 seconds for a top to bottom tilt. Set up a nice scene that tracks people walking or a forklift elevating that has lines in the background, shoot it at 24p with a 180 degree shutter and you have a nice real-world test. Shoot it machine controlled as well as handheld and it would be pretty complete.

With the combination of those scenes and the measured numbers, we can get an idea about how much rolling shutter we can tolerate.

The nice thing about the numbers is that they tell you exactly how much lean or compression/expansion you'll get for a given scene. A lot of specs (MTF, THD, frequency response numbers for 20-20kHz) have numbers that don't nearly tell the whole story. With rolling shutter, the number is simple and definitive. It's a single dimension, point spec: the time delay from the top to the bottom of the image.