Is the sensor being fully utilized?

[Brian Parker]

Does anyone have any concrete information about how the camera is using an 18 megapixel sensor to record a 2 megapixel video stream? Does it record a full resolution (whole sensor) image which the processor resizes down during compression, or is only part of the sensor used for video recording?

We read that the camera has a bigger sensor than most video cameras, but I wonder how much of the sensor is actually being used for the video?

[Jon Fairhurst]

On the 5D, the whole width is used. It's cropped top and bottom to make a 16:9 image. Horizontally, the image is filtered to balance aliasing and resolution. Vertically, the photosites are "binned" (added together), which causes aliasing.

Earlier reports claimed that one of every third line was used, but I recall an interview with a Canon engineer where he said that they were binned. That makes sense, as the noise will be lower with binning than with skipping two of three rows.

From what I've seen, the 7D almost certainly takes the same approach for 1080p, though it's not confirmed. I have no idea what it does to achieve 720p, though I assume that it still uses the whole sensor width.

[Barry Green]

Originally Posted by Jon Fairhurst

Earlier reports claimed that one of every third line was used, but I recall an interview with a Canon engineer where he said that they were binned. That makes sense, as the noise will be lower with binning than with skipping two of three rows.

Can you please point to any authoritative information that discusses how these cameras work? I haven't found any official statements from anyone.

Binning makes no sense on a Bayer sensor, no sense at all. Now, if it was an RGB Stripe sensor, vertical binning would make perfect sense, but on Bayer it just plain wouldn't work.

[Don Miller]

The AA filter strength is determined by the small pixels used in still photos. A stronger AA would help video, but hurt still.
So does red just throw out pixels? That would explain reds worse noise but less aliasing output. When the 5DII was released, Red said something about their superior ability to read/reset the chip. I have difficulty reconciling that comment to Canon reading every pixel every frame.
Also, both Red and the 7D overcrank at reduced resolution (double speed means half the pixels). Is this done by skipping lines?

[Barry Green]

Originally Posted by Brian Parker

Does anyone have any concrete information about how the camera is using an 18 megapixel sensor to record a 2 megapixel video stream? Does it record a full resolution (whole sensor) image which the processor resizes down during compression, or is only part of the sensor used for video recording?

The speculation is that it skips pixels. The theory being that it can't read all 18 megapixels at 60 frames per second, that's an enormous task -- and it certainly isn't likely to read all 18 megapixels and perform demosaic'ing and scaling down to 2mpix (1080p mode) or 1mpix (720p mode) in realtime. Therefore the speculation is that it skips pixels horizontally and vertically to reduce the amount of chip that it has to read.

It's not "windowing" the sensor. It's not using the central 1920x1080 out of the chip or anything like that. It uses the full size of the chip, but perhaps not all the pixels on the chip.

Such an approach would lead to significant aliasing and moire issues, and indeed those have been observed in the video, so it seems like a likely explanation for how the chips are working. I just haven't found any official confirmation that that is indeed how they work. Alan Roberts' BBC assessment showed the effects, but he didn't speculate on whether it was row-skipping, he just concluded that it had a poor downscaling algorithm. You can see the results here: http://www.dvinfo.net/forum/canon-eo...n-roberts.html

Unquestionably the 5D delivers a gorgeous image, with some aliasing and moire. The 7D will probably perform comparably. The BBC chart looks scary, but unless you're shooting for the BBC you probably don't need to stress about it too much; instead spend your time looking at actual footage coming from the cameras and see if they meet your expectations and requirements.

[Don Miller]

Originally Posted by Barry Green

Can you please point to any authoritative information that discusses how these cameras work? I haven't found any official statements from anyone.

Binning makes no sense on a Bayer sensor, no sense at all. Now, if it was an RGB Stripe sensor, vertical binning would make perfect sense, but on Bayer it just plain wouldn't work.

It wouldn't be a straight bayer algorithm. But certainly you can derive a pixel value based on many surrounding RGB values.

I recall discussions about binning, but no direct quote. Binning certainly could make"jello" worse.

I don't know how canon video works.

[Don Miller]

Originally Posted by Barry Green

............

It's not "windowing" the sensor. It's not using the central 1920x1080 out of the chip or anything like that. It uses the full size of the chip, but perhaps not all the pixels on the chip.

Which Red does, yes? Crops for a given resolution?

It is possible to bin pixels on chip, as Foveon does.

[Barry Green]

Originally Posted by Don Miller

The AA filter strength is determined by the small pixels used in still photos. A stronger AA would help video, but hurt still.

Maybe, maybe not. If it was using the entire sensor, yes. But if it's actually skipping pixels, then a stronger AA filter might just result in lower overall resolution. An AA filter is basically a "blur" filter that masks out the highest frequencies and tiniest details, but if the video frame is made using pixel-skipping, then it's dependent on those same tiniest details being reassembled into a smaller-sized full frame, so a stronger AA filter might just result in ruining the overall sharpness of the video...

So does red just throw out pixels?

Well, no, not in the same terms that Canon is suspected of doing; Red reads the entire sensor for its full-frame 4K imagery. Their sensor is purpose-designed for video performance, and is designed to be read at full resolution at up to 30fps.

Also, both Red and the 7D overcrank at reduced resolution (double speed means half the pixels). Is this done by skipping lines?

When going into faster frame rates, Red requires smaller frame sizes, because they window into the sensor. You can get faster frame rates in 3K mode, and even faster frame rates in 2K mode, because the 3K and 2K modes don't read the entire chip. However, they don't use the reported pixel-skipping technique either; they just read a smaller "window" off of the chip. The field of view shrinks in 3K mode, and shrinks further in 2K mode, as it's reading a smaller window out of the chip.

Windowing lets them retain the Bayer pattern of the chip and deliver proper color accuracy, etc.

Binning, would be a perhaps ideal way to accomplish lower resolution and higher frame rates, but it can't be done on a single Bayer sensor. Again, these chips are all monochrome; there aren't color sensors. Now, if you had three chips, you could bin them and get faster frame rates, less aliasing, more sensitivity, and less noise... but with a single chip you can't bin red and blue and green all together into one pixel, because the chips don't record color information, they only record brightness information.

[Barry Green]

Originally Posted by Don Miller

It wouldn't be a straight bayer algorithm. But certainly you can derive a pixel value based on many surrounding RGB values.

Only if you've read all those values. Binning doesn't read them individually, it throws the green in with the red and the blue into a big soup, losing the color value for each pixel. Hence why binning would make no sense on a Bayer sensor.

The speculation about Canon using pixel-skipping, and specifically skipping to every 3rd pixel, is that doing so retains the R/G/B/G nature of the Bayer pattern.

[Barry Green]

Originally Posted by Don Miller

Which Red does, yes? Crops for a given resolution?

Yep.

It is possible to bin pixels on chip, as Foveon does.

Yes, but Foveon is, in essence, a three-chip system. It's just three chips stacked in one place, instead of split by a prism. Foveon is not a Bayer pattern. The restriction on binning is specifically because of the Bayer pattern.

RGB Stripe would also be inappropriate for binning horizontally, but could be done vertically.

[Jay Bloomfield]

This has been posted here before:

Canon : Canon Technology | The Minds Behind the Magic

and it is long out of date, since it talks only about the 1G consumer camcorder, the HV-10. But if you go a few pages into the interview, it gives an example of how an EOS DSLR captures 8 fps @ 8 Mpixels. When you combine that information with the rest of skimpy info that Canon has online regarding it's CMOS sensors, you can get an idea as to the viability of the two "theories" espoused in this thread, as to how the 5d2 and the 7d turn large numbers of Mpixels into 1920x1080 (every third row used vs. binning, etc.).

Canon is not too likely to give out its trade secrets, so this is about all we "consumers" are going to find out, at least for now. What other people have tried is searching for the public information regarding the patents of the various Canon R&D employees, such as:

IMAGE PICKUP APPARATUS AND METHOD FOR CONTROLLING THE SAME - Patent

So far, though, we're still stuck with educated guesses as to how the two Canon HDSLRs actually.

[Jon Fairhurst]

If I can find the source that claimed binning, I'll post it. I saw it about two months ago, and was somewhat surprised.

One clue that leads me to believe that we are seeing "creative binning" is the strong color errors on black and white zone plates. If we were to take every third line and filter it properly, color errors would be minimal. Show every third line, and we would see monochrome aliasing. Instead we see strange color artifacts. Maybe this is due to poor filtering in the horizontal direction, but I can't help but think that there is something more going on.

Straight binning would mean adding values together. It's possible that only common color photosites are added, and that they are appropriately scaled - in the case that you add two sites together, divide the result by two with resistors on the sensor itself. Certainly, skipping two of three rows would be faster and simpler.

Anyway, back to the original poster's question, It's pretty clear that (at least, nearly) the entire width of the sensor is being read, and the vertical image is cropped to provide a 16x9 image. The exact reading structure is not known for sure.

[Daniel Browning]

Originally Posted by Jon Fairhurst

Earlier reports claimed that one of every third line was used, but I recall an interview with a Canon engineer where he said that they were binned. That makes sense, as the noise will be lower with binning than with skipping two of three rows.

But the noise isn't lower. In my (unscientific) analysis, the noise looks about the same as taking a JPEG and downsampling it with a nearest neighbor algorithm. It is much noisier than taking a JPEG and downsampling with sinc or any good algorithm.

It's worse in photon shot noise dominated image tones (midtones at low ISO and highlights at high ISO) as well as read noise dominated image toines (midtones at high ISO and extreme shadows at low ISO). If it was binning, the photon-shot-noise-dominated tones would have the same noise and the read noise would be lower than just resampling.

Binning a Bayer CFA sensor would likely result in certain artifacts, but not nearly as bad as what the Canon video has. There are some known techniques for reducing the artifacts in real time, Phase One is probably using them in the binning mode they added through a firmware update a few months ago.

Canon's sRAW2 is more in line what I would like to see with Canon video: the noise and aliasing are worse than a downsampled still, but not nearly as bad as the video currently is. But sRAW only needs to do 4 FPS, not 30.

[Jon Fairhurst]

Here's a credible source: Tim Smith, Product Manager of Digital Imaging at Canon.

Digital Cinema Society - New Streaming
(Go partway down the page. There are two QT files of Tim Smith. It's from July 23rd.)

In part 1, about half way through, Tim Smith strongly recommends UDMA cards. He also confirms that they don't pixel skip. They bin them in groups of six to pull down to 1080. He also says Keeping the camera level helps.

He talks about the need to re-clock the chip to get to 24p, and says its unlikely, but not impossible that there could be a firmware update. Then again, he would have bet money that they wouldn't release a manual control firmware update.

Tramm got Canon's attention.

Canon is thinking about cine-style lenses. Some of the development group silos are falling down.

Part 2 shows a Ghost Whisperer webisode, shot back in the auto-only era. He talks about his experience on that shoot. He mentions the 50L, 85L and 16-35L.

Tim recommends leaving HTP on. (I don't, unless the dynamic range of the scene is extreme.)

He's seen the 5D with a PL adapter. He hasn't yet seen a film transfer. When projected large, he expects that you will see lens flaws with cheaper glass.

The best news is that Tim Smith "gets" the needs of videographers. It's not like he's a photo guy who is just now being exposed to video and film.

Anyway, the videos are worth watching for anybody interested in the development of HDSLRs.

[Chris Barcellos]

Jon:

Very interesting video... First acknowledgment by Canon that I have seen that Tramm's work has caught their attention.

I am not sure how technically competent he is, as he seems to believe file size can be increased by a hack. My understanding is that file size (at least in HiDef) is limited by the 4 gig maximum size. I think the reference to the guy in the guy is Seattle not having hacked that is kind o f interesting, as they seem to be mixing your location, somebody in Europe and Tramm as an all in one deal.... maybe he is referring to linked files when size limit is over reached.

He is pretty clear that 24p won't come out of chip without clock change, but then again, something that technical isn't his forte. Most interesting is the acknowledgment that Canon never saw this camera gettting anywhere near film making...