Camcorder comparisons?
 

What kind of tests would you recommend for camcorder comparisons ... how would you setup the tests?

I was thinking we could come up with a list of tests that are rigorous and repeatable. Anyone could run the tests on their camcorder and post the results online so we can compare the results.

Example tests: 16:9, low light, image clarity/sharpness, color fidelity, etc.

This site shows new cams with their basic specs. Plus links to pic comparisons.

http://www.grandeye.com.hk/etx/900/index.html[list=1][*]16:9 - only some cams have good 16:9 - and we know which ones these are[*]low light - you would have to set lux/test guidelines[*]sharpness - we would need wave-form monitors[*]color - I for one am color blind. I'll be happy to oblige, though. :-))[/list=1]

I think this is a great idea. The biggest problem obviously is the inability to shoot the same subject under the same lighting conditions. Shooting resolution charts really wouldn't accomplish much. If everyone had light meters, I guess we could all contribute shots of a room with the exact same level of ambient light at a various levels for one test. That might tell something. I have a GL1 and would be glad to participate if other people are interested. Maybe we could come up with an object that everybody could buy locally and shoot under tungsten bulbs and daylight. Or else we could mail an object back and forth, but there are clear problems with that. For one thing, it would take forever. Given the number of retail chains out there, surely there's some kind of colorgul widget we could all find locally.

How about the strength test? Drop 'er cam from 1 foot and see if she still works; then from 2 feet, etc. :-))

PS: and don't exclude these fine 3 tests: heat test, cold test and 'er humidity test. ;-))

http://babelfish.altavista.com/babel....html&lp=ja_en has lots of camcorder comparisons.

Shooting things which we know what they're supposed to look like might work. Trees, people, etc.

What's a good way to show the quality of the 16:9 mode? Would having standardized shots work? Maybe use a close up with a medium shot etc. Maybe add in some fast panning or bright lights to see how it behaves? making it repeatable would be the goal so that we could compare the same shot/video from camcorder to camcorder How do you set this up? Is there an common instrument that measures lux? okay i'll have to research this first
That settles it your in charge of the color tests

Some more tests:

Handling blown out highlights
Grain at different light levels

measuring sharpness would kind of only require one person with a wave-form monitor, since we can transit DV stills or short clips over the internet with no loss. However, I don't think a wave-form monitor is necessary as long as the *methodology* is consistent. This would give meaningful, comparable results. You could just hook your camera up to a computer and view on a computer monitor, which have excellent resolution.

However there is a small problem with sharpness tests in that it can be affected by the Kell factor. You have to pan your camera in sub-pixel increments if you want to measure lines of resolution off a resolution test chart (like the EIAJ one). You also have to frame the test chart very carefully, and print it out at resonably high resolution. However you only need to do all that if you want really accurate results.

Michael, you'll drive yourself nuts. It's easier said then done.

When I was researching which camera i should buy I made a test tape and took it with me to any camera store i went to and shot as much comparison footage as possible.

If a store had 3 cameras that I was considering I'd tape footage of all 3 , same subject same light. if the store had 2 i'd do both. it's hit and miss.

Yup. That's the way to do it: take a fresh Fuji miniDV tape with you to the cam shop, then take test footage of everything. The salesperson will love ya. :-))

As someone who views images all day long, hundreds of images, sometimes thousands, in my role as a photography teacher, let me say you're missing the most important part. The subjective nature of our medium and finished work makes picking cameras extremely difficult when comparing performance specs.

I have looked at cameras and used cameras that are supposed to have very sharp images, but they don't look as sharp to me as other cameras (which are supposed to be less sharp). The same with color and low light and many other aspects of camera performance. I've had clients say what great low light images they got with their camera. But to me the image was worse than many. The image was very noisy, but the client liked the greater amount of color and the noise didn't really bother him. I would have rejected his camera based on the noise, but he loved it based on the color.

Other subjective features weigh heavily in our selection and use of cameras. I will sometimes pick a lower end camera to shoot with because it will have better performance in key areas. I've shot with cameras that might have better focusing capability, better viewfinder, better audio, over a camera that might have a slightly better picture.

A project I worked on several years ago gave me the opportunity to shoot with two different cameras. I choose the model with a slightly lower quality image, but a much better viewfinder. Why? The project called for a lot of run-n-gun style shots. It wouldn't be possible to use an external monitor and probably little opportunity for retakes. The better viewfinder meant I could make sure the image was in focus (sharp as possible) and maximize the potential for using the footage. The camera with the potentially better image wouldn't have mattered if a lot of the scenes were not in critical focus. Besides in most cases you can improve the image in post.

I really think that in picking a camera each user should prioritize their needs in a camera and try to match the camera to their needs. Shopping and buying the camera with the best specs can lead to serious disappointment. Everyone's needs are going to be different but you could establish some categories such as, low light, high contrast, etc. that might be helpful. But the difficulty with this method is how the comparison is done. For example, one camera might do good in low light situations right out of the box, with no user adjustment. But another camera gives only fair results in low light, right out of the box. But if the second camera is adjusted properly it gives a superior low light image to the first camera. So how do you evaluate and rate those cameras? Do you test and rate them as if you're a novice user and know little about your camera (having never read the manual) or do you test them setup and tweaked to the max? If you test and rate the tweaked version the novice may be disappointed that he can't get the results the testers get.

Many cameras today are so very close in performance that the differences don't matter, or can easily be adjusted via in camera menus or corrected in post. Then how should a camera be picked? At the very least Bryan and Frank's method will beat any chart of performance specs. But if you ask any of the really experienced shooters, Chris Hurd, Charles Pappert, Wayne Orr, Bill Pryor, many of this forums wranglers and dozens of others here I've missed, about how to pick a camera, they'll tell you to get to a big show or convention. Go to NAB, CES, DV East, DV West, PMA, state PPA conventions and the dozens of state and regional shows and compare models. Pick up the cameras and play with them. That's what they are there for. You wouldn't buy a car without a test drive, why would you buy a camera without shooting with it?

I know many people's arguments are they live in a rural area and can't make it to the big shows in the east or west. Then consider many of the state and regional shows sponsored by organizations such as the PP of A, or various wedding organizations. You won't find Sony or Canon there but the bigger regional dealers will be there with cameras for you to compare. If you're considering a bigger ticket camera, say $3,000 and up then you really need to go to a bigger show, if you can't find the cameras locally. It's a couple hundred dollars for a plane ticket and another hundred or so for a room and meals. Compare that to spending $4000 or $5000 on the wrong camera. The money spent may delay your purchase a month or two, but nothing like the frustration you'll feel when you discover that you spent $4000 on the wrong camera.

Why I often rely on only test results
 

I totally support Jeff Donald's suggestion that hands on tests are very desirable. Unfortunately it's not always practical in my experience. When I was in the market for my first camcorder recently, the camera store I had insurance credit with did not have any of the models I was interested in on display. Actually nether did most of the other stores. In my experience, when I go to trade shows, they just happen to not have the particular model I am interested in. As I live in Canada, attending tradeshows in the US is also bloody expensive, we have to pay for things at our 2nd tier world dollars. That is why I think tests, and user feedback on forums such as this are highly desirable, even though I agree we should make every attempt to test these things in person.

• You're right, but I figure it's the lesser of two weevils.
  
  Either way, it's driving me nuts.

The tests I'm proposing are just supposed to be part of the process of buying a camcorder. I'm assuming people will do their own hands on tests **and there's not much we could do if they didn't

Right now though the process can be so subjective - ask about low light and you'll get 10 different answers. A pro will say you "have" to go with the VX2000 if you want low light, and then an amateur will say the Optura Xi's low light is "great" ... and they'll both be right.

So I'd love to have comparable video tests so we can decide what's acceptable. People could also run the tests on their own camcorder to see how it compares.

FYI: The results would be a video file/s taken in similar lighting condtions using similar settings (auto or manual). Anyway, it looks like I'm going to have to do more research and come up some suggestions before this proceeds further.

Frank - I'll ask B&H Photo about the "drop" test ;-p

I was also thinking of a spill, smudge, little dirty fingers, big fat finger, and cold wet nose test.

Standardized tests are needed
 

I think it is great what you are proposing Michael. Comments are generally too subjective. We need standardized tests. Robin Liss at CamcorderInfo said she was going to do such tests, but they ended up being too simplistic. And your suggestion to make a standardized test so individuals can even do the test is the best suggestion I have heard so far. Here are some things I would like included in the tests and evaluation. But these should be video and not still tests. I'm still adding to these:

PICTURE AND SOUND
(1) Picture brightness and clarity as % of normal at 1, 10, 100, 1000 LUX, using readily available standardized objects or scenes.
(2) Grain at 1, 10, 100, 1000 LUX
(3) Dynamic range of colors at 1, 10, 100, 1000 LUX
(4) Sharpness
(5) JPG compression, inter-line twitter and other artifacts
(6) Color correctness compared to test sheet at 1, 10, 100, 1000 LUX
(7) Lense flare and aberation
(8) Resolution
(9) Sound quality, stereo available, motor noise, wind cut, jacks, hot shoe microphone

CAMERA (major features)
(1) 1CCD, 1CCD RGB, or 3CCD
(2) Manual controls available, focus, shutter, aperature, sharpening etc.
(3) Stabilization, OIS or DIS, and effectiveness
(4) Bottom or top loading
(5) Optical zoom ratio
(6) Mass and size
(7) Photosite area
(8) Ratio 4:3 and/or 16:9 with photosite to recorded pixel ratio
(9) Viewfinder and LCD viewability and details
(10) Interlaced and/or progressive mode available
(11) Environmental evaluation, shock, water, cold, heat, handling. I wouldn't want to damage my camera or a borrowed one, but in case people do drop their camera by mistake they could report this.

Cool! Thanks, that's a great list

Ben you are kidding, aren't you. I rather doubt most profesional reviews go to that much trouble.

How come you didn't do all that with your 852?

Trouble is, few, if any present day cams don't even come close to our expectations, unless one replaces the PV-DV953's CCDs with the DVX100's CCDs, drill a hole to add an XLR socket and spray paint the shell black. :-))

My short list
 

Bryan, professional reviews don't include tests of all the above. That is why we are left guessing. The above list is only my short list. And no I didn't check all the things in the list before buying my 852 but I had most of them in mind. I can't do all these tests myself and no one had produced a report like that for my camcorder.

"Bryan, professional reviews don't include tests of all the above. That is why we are left guessing."

I'm not left guessing. I know my equipments capabilitie's, it's limitations and it's shortfalls and I make do. Put a fully charged battery in the camera and a tape and let's see what it will do. Same with the audio.

Now if we were talking a several hundred thousand dollar camera i could see some of the information would be valid, but a consumer or prosumer camera. Who would want to waste the time even trying.

There are those that would waste a half a lifetime selecting the perfect widget only to find that it didn't really exist.

To some that is a hobby in itself. I suspect that you enjoy testing a product more than actually using it for the purpose it was designed for and i'll wager that you thoroughly enjoy yourself doing so. There is no negative connetation attached to my statment, other than to say that we don't all have the same need for perfection.



This i

Who is supplying, or purchasing, the tens of thousands of dollars of test gear required for some of the tests you purpose? Even the $200,000 cameras Bryan mentions, aren't put through reviews like you purpose? Why? Because the people who would buy and use those type of cameras will want to shoot with the camera and subjectively evaluate the image quality.

When I used to purchase cameras for my production company, I rarely looked at the technical specs. I reviewed the list of features to see if the camera had my required feature set. But the technical specs like signal to noise ratio of 60db vs. 59.9db made little difference to me. The picture quality was much more important than a .1db difference in a spec. The subjectiveness of image quality can't be condensed to a single number or even a series of numbers or test.

dpreview.com
 

Phil Askey at dpreview.com has created an extreemly popular website for digital still cameras. He tests about as many equivalent things as I was proposing. People flock to this site to get reviews. Thousands of people all around the world choose camera based on these extensive tests. Even $600 cameras. There are thousands of digital still cameras. Who would get to test most of these in a camera store or show? There are also too many camcorders out there to be able to always find one to test in person, in my humble opinion.

I can see an argument for a standardized review. Many of the reviews are simply somebody regurgatating the manufacturers promotional information. The review criteria you suggest is extreme overkill though especially for consumer , prosumer cameras.

It would be an excellent make work project if someone were to obtain funding. Sounds like something the Canadian Government would go for.

If such a review was mandated then think of the consequences. the price of the camera would be increased and their would be a need for books and even seminars to decipher the findings. Chris would have to add several new forums.

Nah, I'll just flip a coin, thank you very much.

Yeah dpreview.com definitely sets a good precedent for this and Ben's list is a good place to start.

Bryan's got a good point, those tests might be a costly and time consuming process but I think if I start with the highest possible standards and then work down from there to the most practical and easiest tests to conduct we can get some good information for comparison - and we'll end up with a lot of good eyeball material for people to judge themselves.

Manufacturer technical spec comparisons are easy enough, in fact I'm almost done with the spreadsheet, for what it's worth, and I'll post it in about a week. So far I've got all the current camcorders for Sony, Panansonic, Canon, and JVC specs matched up, but I was having problems with the higher end cams (above US$5,000) because I didn't understand some of the specs.

Fipping A Coin
 

Actually I'm open to many things. Bryan has a suggestion that I've not thought of before. Tomorrow, I'll consider checking it out. I'll flip a coin to see whether I should even get out of bed, then whether I should eat breakfast and so on. It might lead to some interesting things actually.

Ok
I give up, but then i have a camera and equipment that i'm happy with. I responded with the coin toss because I thought it better than incurring a masssive debt for a questionable review protocol.

It would be better to ask a happy customer how they made their decision rather than one who is unhappy. Jeff has had years of expereince, operating as well as overseeing the repair of video cameras, yet his methods for selection are simple.

It happens all too often that we get mired in an unrealistic world of specifications that have absolutely no baering on our actual usage of the product.

How does it look, how does it sound, is it dependable and well built? Will it do the job that I want it to? Is it reasonably priced?

It works for me and I am a tech, everything I do and enjoy has a technical base. Why make things more complicated than they really have to be.

I'm flown all over North America to find the answers to problems that others can't solve. Most of the problems I find are very basic. People have overlooked the simple reason and made the whole process more difficult than it should be.