View Full Version : Beware of this invalid camera test


Manus Sweeney
March 3rd, 2010, 02:36 PM
Comparison tests of the Canon 5D Mark II, Sony PMW-EX3, RED One, Sony F35, and Arri D21 (the end of the clip is easier to see the comparisons)..

Maybe the picture profiles werent dialled down to a flat setting but its still pretty shocking!

Ben Cain - Negative Spaces - Michael Ballhaus: Over and Under Exposure Tests (http://www.negativespaces.com/blog/2010/2/25/michael-ballhaus-over-and-under-exposure-tests.html)

Chris Barcellos
March 3rd, 2010, 02:59 PM
I assume what they label as D5 is the 5D Mark II.

I am not sure what that was all about since no sound was provided, but I don't think there is any surprise that video has less latitude than film..... Our challenge as film makers is to work within the medium we have.

Bill Grant
March 3rd, 2010, 04:12 PM
I'm so glad I couldn't tell the difference between any of them. I was very excited to see that the 5D looked almost the same as the rest.
Bill

Manus Sweeney
March 3rd, 2010, 04:16 PM
i guess you didnt watch the clip till the end!?

its the side by sides that were quite scary.. im sure theres many variables and reasons why it could have turned out so bad but i was curious to see what people think about it

Stephen Mick
March 3rd, 2010, 04:19 PM
Tests like this make me glad nobody watches movies in the "side-by-side-comparison" format. An interesting test, certainly, that changes nothing about how I work.

Richard Gooderick
March 3rd, 2010, 04:52 PM
I'm not even going to watch it.
Winter is too cold. Summer is too hot. The glass is half empty.
Life is too short.
;-)

Chris Barcellos
March 3rd, 2010, 05:49 PM
i guess you didnt watch the clip till the end!?

its the side by sides that were quite scary.. im sure theres many variables and reasons why it could have turned out so bad but i was curious to see what people think about it

we don't even know how they processed this stuff. 5D footage processed through Cineform recovers detail at both end of the spectrum, that Canon's codec preserves. So this is a real piece of junk "test" apparently sponsored by film sellers, from what I can tell...

Peer Landa
March 3rd, 2010, 06:53 PM
we don't even know how they processed this stuff.

Doesn't even look like they white balanced the 5D.

-- peer

Manus Sweeney
March 4th, 2010, 03:30 AM
im pretty sure its a difference in saturation rather than in white balance.. i guess a DP with a cv like this wouldn't forget to white balance! (even if he didn't know how to flatten the picture profile) Michael Ballhaus (I) (http://www.imdb.com/name/nm0000841/)

personally im new to DSLR video (and a big supporter) and not long in video at all but i do like to follow developments in camera technology. For me the result of watching the test wasnt to say that HDDSLRs are bad, but its maybe a gentle reminder that there is a reason why an EX3 costs 4 times as much as a 5d and a red costs 4 times as much as an EX3.

For those that arent interested in these kind of comparisons, my apologies for posting but personally i found it quite eye opening and was genuninely curious about opinions on the results and the validity of the test..

Nicholas de Kock
March 4th, 2010, 09:26 AM
This test is quite flattering for Canon. Their consumer 5D tested along side the Sony F35, Arri D21 is a scary thought considering that the Sony F35 & D21 costs over $100k! The test is rather useless though, the only thing that really matters is what the image looks like properly exposed any over exposed footage is garbage on any camera. Having worked with the 5/7D it's clear that the 5D was not flattened out, looks like factory presets. Once again the 5DII is a $2500 camera compared to cameras over $100 000! Shocking!

This test displayed in cinema would look totally different I bet but on the web all cameras are equal.

Nigel Barker
March 4th, 2010, 09:26 AM
For me the result of watching the test wasnt to say that HDDSLRs are bad, but its maybe a gentle reminder that there is a reason why an EX3 costs 4 times as much as a 5d and a red costs 4 times as much as an EX3.An Arri D21 or Sony F35 without lenses costs more to rent for one day than the 5DII costs to buy outright. Clairmont - HD/Digital Cameras (http://www.clairmont.com/catalog/us_cat/pages/digital/digital_camera_acc.html)

Nicholas de Kock
March 4th, 2010, 09:50 AM
There is something "very" wrong with those side by side comparisons. Whoever did them must think we are complete idiots. The side by side on that Vimeo clip is a complete fix, I took screen shots of every exposure stage and compiled my own comparison. The saturation on the 5DII is not even normal, someone really cocked up.

Manus Sweeney
March 4th, 2010, 10:03 AM
well spotted!

certainly something fishy going on there.. ok i guess we can put an end to the thread and (for whoevers interested) wait for Philip Bloom and co's film vs HDDSLR results next week!

Ian G. Thompson
March 4th, 2010, 10:23 AM
This test is crazy off.

Jim Giberti
March 4th, 2010, 10:55 AM
First, For those of you that might be tempted to watch this - you'll be better people for not doing so.

Second - there are reasons why it's a good idea to have a pretty woman or at least a still life when doing a comparison test.

Manus Sweeney
March 4th, 2010, 11:15 AM
as i mentioned im new to the DSLR world (7d).. so far i have noticed that ive missed exposure a bit sometimes and in those cases the highlights or shadows are really hard to pull anything out of (compared to video cameras i've previously used)

i think that will influence a decision to get an ext. monitor sooner rather than later, and also to flatten the contrast settings even further down.

as for the 'test'.. it is really bad that they can publish something like this and the fact that the guy whos name is behind it is someone responsible for 'Goodfellas', 'Dracula', 'The Departed' etc etc.. I'm sure I'm not the only one who believed the results.

Jim Giberti
March 4th, 2010, 11:26 AM
Then it sounds like we should be talking about what settings you're using and how you're transcoding and editing, because we're getting great latitude and shadow detail with a basic setup of Neutral>0 Sharpening>0 Contrast>-1 Saturation. We drop the clips onto a Pro Res droplet and voila, great footage ready to grade.

Quick example I mentioned recently. We did a short in January with a Blackhawk helicopter in a snowy valley in brilliant mid-day sun...pretty much the holy grail of tough exposure.
Everything was shot either through a Hoodloupe on one or with the LCD on another and there wasn't a blown highlight or crushed black to be found.

Ian G. Thompson
March 4th, 2010, 11:27 AM
But at the risk of sounding like some Canon fan boy I think there are so many things wrong with this comparison …I see some of them were mentioned already. But, the footage coming from my 7D does not look like this. There are so many things one can do to combat this “over-contrast(y)” look that seemed to come from the 5Dll during this test. One…there are many …many custom flat gamma curves out there that you can upload to your camera. Second…”Highlight Tone Priority”….it works. It actually gives you one whole stop of detail in the high’s and at the same time raises the blacks (in some cases at the cost of extra noise). But my point really is if the operator worked to get the flattest picture from these DSLRs then the results would have been a lot different (IMO). It’s not to say that these cameras don’t have their weaknesses…sure…this test is a good way of (for lack of a better term) “exposing” them but I believe there’s definitely something amiss here

Nicholas de Kock
March 4th, 2010, 11:30 AM
Manus don't take offense by the comments I'm glad you pointed out the test. I also struggle to expose properly on the 7D, it's very hard to judge proper exposure. I'm sure the Red, D21, F35 blow the 5DII out of the water for cinema but for everyone else the 5DII is awesome.

Ian G. Thompson
March 4th, 2010, 11:37 AM
Exactly....given how much more they cost those cams should blow the 5Dll out of the water..even for this test. I'd expect that...but from what we see here the look achieved by the 5Dll seems off compared to what we've seen elsewhere.

And yes...Manus...thanks for posting this. I think this sort of stuff is cool to see.

Jim Giberti
March 4th, 2010, 11:58 AM
No offense Nicholas but why would you be glad to see a test that is so obviously devoid of meaning. That "test" was in no way indicative of anything that the 5D produces by anyone who knows how to shoot with it, so it's not a comparison - it's misleading and Manus acknowledged such.

Comparisons like this give people false information regarding camera choices and when the thread starts out with a provocative title like " 5D showing it's weakness" then it tends to get attention...no?

As has been pointed out many times before, this is a technical information site and if the technical information that's being shared is obviously misleading then it needs to be set right.

As it was shown, the stills used for the conclusions of the "test" don't even match the footage shot. It's got nothing to do with Manus, he posted it with all the best intentions I'm sure, but it's still misleading, which is why I tried to address his concerns from a realistic perspective of settings and transcoding - so that as a new DSLR shooter, he has useful technical information.

Ilya Mamonov
March 4th, 2010, 12:17 PM
Well, I was a sceptic about 5DII abilities. But the more I learn about it the more I see. Obviously those tests were fixed. What's the reason to fix the test unless you are affraid of what 5D is capable of? Very interesting.

Jon Fairhurst
March 4th, 2010, 12:29 PM
as i mentioned im new to the DSLR world (7d).. so far i have noticed that ive missed exposure a bit sometimes and in those cases the highlights or shadows are really hard to pull anything out of (compared to video cameras i've previously used)...

Manus, you're right. It's really important to get a good exposure - especially for face tones or any item important to the story. If the exposure is too high or low, it pulls the item of interest out of the linear area (where you have lots of bit depth) to the ends of the S-curve, where things get shallow. Since we're in eight-bits, pulling detail out of the ends of the S-curve isn't possible.

An S-curve is great for keeping features that would otherwise be digitally clipped, but it doesn't give you a lot of latitude in post. Good exposure with DvSLRs is critical for getting great results.

Buba Kastorski
March 4th, 2010, 02:25 PM
I can't see much on original 640x360 clip, but I really like what i see;
the least expensive from compared to is 3X of 5D, (price wise)

Matti Poutanen
March 4th, 2010, 02:51 PM
A thought: could the very weird looking 5D image in the end of the video (in the comparison shot with picture from all of the cameras) have something to do with the fact that the 5D operates in RGB color space, as every other camera in the test does Rec709 color space?

Jim Giberti
March 4th, 2010, 03:14 PM
Honestly my reaction is based on one thing - you never know the experience level of people who may see these things, internalize them and then repeat them to others and in other forums.

Some people see things like this on a credible site like dvinfo and assume that it's a valid test performed under controlled circumstances with a knowledge of all the cameras involved and how to set them up appropriately for accurate comparison.

Anyone whose ever been involved with real tests/shoot-outs/comparisons like this knows how much is involved (usually several technically skilled people) in getting it right.

There are so many examples, in so many venues, of exactly how the 5DII looks when setup and shot well that it all seems pretty counter-productive.

Being technically accurate and informative - this is not the "5D showing it's weakness". A good name for the thread might be "How do I adjust my 5DII so it doesn't look like this?"

Silton Buendia
March 5th, 2010, 05:42 AM
Seriously people this is a fix and if you don't know that then you don't even belong doing any kind of video production.

First off the whole purpose of this stupid little video was to blow some new Kodak film stock. Film is dead, they are trying to justify that its not. More and more hollywood movies are being shot digitally and they are coming out just fine, the general consuming audience doesn't have a clue.

Then there are clips and stuff in this test that just don't ad up. I opened two windows side by side and it amazes me how different the 5D and EX footage look in the beginning in comparison to the footage at the end of the video. Its like night and day, interesting that its so different. Who ever posted this video is a follow for thinking some of us are such idiots to believe this crap.

Chris Hurd
March 5th, 2010, 07:59 AM
Some people see things like this on a credible site like dvinfo and assume that it's a valid test performed under controlled circumstances with a knowledge of all the cameras involved and how to set them up appropriately for accurate comparison.You are correct in your evaluation, but not in your execution.

The proper way to handle this sort of thing is to use the Report Post function (it's the little "!" icon to the left of any post). We'll then make whatever changes we need to in order to prevent things like this from being mistaken as something we endorse. For example, in this case I'm changing the title from "5D showing its weaknesses" to "Beware of this invalid camera test." Closing it as well.

The last thing we want here is for people to start getting ugly over it, which creates more work for us to do with regard to thread management. Remember, all we're interested in on this site are the technical and creative details. If you have something to say that you feel like you really need to express that isn't specifically about the technical and creative details, then I encourage you to start your own blog / forum / whatever and say it. Thanks for understanding,