4:4:4 10bit single CMOS HD project

Obin Olson · July 17th, 2004, 05:08 PM

when will we have pci E frame grabbers Steve? that is a VERY fast standard Shuttle has a new board that has one pci E slot..news??

Juan M. M. Fiebelkorn · July 17th, 2004, 05:26 PM

Obin and Steve,
So If I add a mechanical shutter at 1/48th to the sensor and run it at 24 fps (not 48 to avoid rolling shutter) I won't have the smearing problem?
I ask cause, if this solution gives better quality I'd go this way.

If someone is interested DCRAW uses the Variable Gradients Method.

Ben Syverson · July 17th, 2004, 05:40 PM

Variable Gradients is an interesting approach, but from what I understand, it's fairly slow.

My algorithm could probably work in a realtime pipeline if you optimized it. Processors with SIMD (single instruction multiple data) such as the G4/G5 and the newer Pentiums should have no problem with it. Also, I think it could be put onto a FPGA pretty easily if that ever becomes a reality...

Hey, I posted the PC version of my updated linBayer plugin here for you guys to check out. Jason + I have been working on getting all the kinks out, and I think it's to a pretty good point. I was able to eliminate some artifacting in noisy areas in this latest build.

David wrote:
We don't have any Macs or a development enviroment for a Mac. At least for today all CineForm tools will be PC based.

Sounds like you're missing out on a massive chunk of the market (if you're at all interested in this market). Mac users probably make up the majority of people interested in low-cost HD solutions for filmmaking.

Just a thought. $10k investment for a possibly massive return...

Les Dit · July 17th, 2004, 08:13 PM

Might be the opposite.
People willing to play with non standard and somewhat self made hacked together hardware are more likely PC people.
Ever been to a Computer swap (fair) ?
Here in LA it's like two football fields of vendors with all shapes and forms. No Macs, except a couple of old junk ones in a used pile for $5.

Maybe people who hate being locked into a single source for the computer also eschew the big name video vendors, like Sony and Matsushita.
I like open hardware, myself.

Jason Rodriguez · July 17th, 2004, 09:03 PM

That's okay, we Mac users replace not having open hardware with open software ;-)

I do think it's pretty cool that since OSX/Darwin has hit the market, I've been able to play around with all the goodies of the OSS market while still enjoying "mainstream" apps like Adobe, FCP, etc.

I'm sure you can do a lot of similar stuff with cygwin, but I've found many GTK-based stuff, or anything that wants X-windows to be pretty buggy on Windows versus a native X interface like I can get in OSX (special install).

Also Ben, I bet if you wrote that linBayer as a CoreImage/CoreVideo plug-in, you'd be seeing real-time results. Of course that won't be released till "Tiger" comes out, but I've seen it do real-time gaussian blurs, etc., and those filters take a lot longer to process on my machine (dual 2Ghz G5) than your linBayer filter does in AE-so I'm assuming that it's less processor intensive than a heavy gaussian, and if that plays back RT in CoreImage/Video, then your filter should be a real-time conversion as well.

P.S. BTW, I'm not trying to make this a Mac/PC war, I realize that right now I'll have to purchase a PC anyways if I'm gonna have one of these industrial cameras.

Obin Olson · July 17th, 2004, 10:38 PM

OMG CoreVideo looks HOT!!! I hope someone does this for windows soon...we are all using $100 agp cards that could do EVERYTHING in realtime even HD ..yet not a program uses them but games...shame...it would be Apple that changes that ;) and I am a windows person

Ben Syverson · July 17th, 2004, 10:58 PM

Les wrote:
People willing to play with non standard and somewhat self made hacked together hardware are more likely PC people.

Yes, most definitely. I'm talking directly to the vendors: if you wanted to, you could develop an HD system that didn't need to be "hacked together," one that could be a much more mainstream solution. Think of all the FCP jockeys out there begging for a better format than DV or HDV. The filmmaking market is at least as big as the scientific market -- trust me. And a lot of those people are Mac users.

I can name 10-20 people off the top of my head who would put down a deposit right now if you could deliver them a camera that pushed HD footage at 24fps through USB2, Firewire or Firewire800. You could sell such a camera for $3000 or $4000 (with $1000 - $2000 worth of parts) that would appeal to thousands of videographers, filmmakers and others for whom HDV is not going to cut it. But CameraLink will not sell units to this crowd, except a few dedicated individuals like Obin.

This whole thing is set to explode in the next 18 months, and whoever is out there first stands to gain the most ground. Here's what you need: a 720p camera that can deliver at least 24fps over one (ideally a couple) of the interfaces I mentioned at 8 or 10bit. 12bit is overkill for the most users. 10bit 720p at 24fps is around 26MB/sec, which most 7200rpm drives can keep up with. Maybe do some compression via a FPGA in the camera to get it down to 18 or 20MB/sec -- by the time you have a shippable product (12 months?), you'll have no trouble approving laptops to handle it.

Software vendors are screwed -- this market won't like kludgy proprietary tools. We'll build our own or buy better integrated tools.

In the next 18-24 months the "industrial" camera websites we all visit will cater to both the scientific and filmmaking communities. Some may only cater to the film community (since it's potentially far more profitable).

Be there first.

- ben

Obin Olson · July 17th, 2004, 11:07 PM

I was here first ;)

looks like windows xp embedded is a good os for the camera

Ben Syverson · July 18th, 2004, 12:53 AM

Obin: We all give you props for being our Chuck Yeager. :)

Jason + I ran into an interesting problem today. He noticed a "gridlike" effect in the de-bayered results of my filter, when I knew there was no reason for that effect to occur normally. I looked into it, and we found that the green pixels in odd rows were often globally different than the green pixels in even rows. When I analyzed the pixels, I found that the green pixels in a blue row might be (in 8bit):
100, 98, 99, 97, 96, 98
but on the next row, a red row, the green pixels might be:
92, 90, 90, 89, 85, 88

Steve, have you seen this before?

The rows were sometimes off by as much as 10 or 12 values, which is a significant deviation. The only logical explanation for this is that the physical bayer filter grid on the sensor is veeeeeeeeery slightly mis-registered. Like it's a little up or down of where it should be. As a result, the blue and red filter areas are covering up the green sensors a little bit, and since blue blocks more light than red, you get a value difference. The red+blue channels come out even because they're being corrupted equally by only the green filters. But the greater the registration error, the lower the accuracy of the color. I verified this hypothesis by analyzing pixels from areas where the R & B are very different, and where they're almost the same -- sure enough, the greater the contrast between R&B, the more "interlacing" artifacts in the G.

This isn't Micron-specific, I've seen this on images from the IBIS5 sensor as well. It must be a manufacturing nightmare to register a filter on a 2/3" megapixel + sensor, when your margin of error is around 1 micron -- 1/25400th of an inch! So guys, this is yet another reason to buy sensors with bigger pixels -- not only does their light sensitivity go up, but the bayer registration is almost certainly better. If you have 12 micron pixels as opposed to 6, an error of 1 micron won't be so dramatic.

I'll build a slider into linBayer to counteract this effect on the green channel, but using it will necessarily affect the original values of the image, and thus the process will not be totally lossless.

- ben

Juan M. M. Fiebelkorn · July 18th, 2004, 01:29 AM

More cameras:

http://www.mikrotron.de/en_cameras_m...n.shtml#mc1301

Ben Syverson · July 18th, 2004, 03:58 AM

After playing with more of the Micron images, the bayer registration seems to be pretty solid. It's the IBIS-5 images that need more help.

I posted the updated version with registration correction, and more control over the de-zippering process.

Steve Nordhauser · July 18th, 2004, 08:36 AM

Ben on zippers:
The Micron has 4 analog gain amps - R, G1, G2, B. This means odd and even rows have their own gain setting. Either they are set differently or they are not balanced in the original image. For Obin, I would suggest live viewing and zoom way up on a solid color image. That is how we balance columns on the IBIS5.

I haven't heard a FG manufacturer talk about PCI-E. PC-104+, yes.

Ben on Bayer:
I think you hit something very significant when you said that the original pixels are not modified so you can go back at any time and re-do the Bayer. I hadn't thought of that before. It means that if you find an incredibly complex algorithm, you could run it on the final cut of the film if you stay had high bit depths through editing.

Smearing:
In about 1.5 weeks I'll let you know about the SI-3300 smear - micron says it should be much better. I haven't heard of the Altasens smearing any which way.

Obin Olson · July 18th, 2004, 03:38 PM

Steve:

I don't see how we can use the 3300 at all as your going to window the chip for 1280x720 and even for 1080p..it's a 1/2 inch chip as-is so making that chip work with a decent FOV...I don't see how unless you can do some type of superpixels/binning ?? OR and this is the BEST if you could resize the native image BEFORE it is saved with a realtime dsp/fpga chip?? if you could resize it would make that chip BETTER then the Altasens chip because it will be MUCH sharper @ the resolution of 1080p or 720p because it's been downsized..this would be VERY good...

If that could be done I would take the 3300 over the Altasens

Juan M. M. Fiebelkorn · July 19th, 2004, 12:40 AM

About SI 3300,

Isn't it a 1920x1080 capable Sensor??
Am I missing something?

edit:Oh, sorry.I went to SI's page And I read it is a 2048x1536 chip but at 15 fps.
Does SI have a chip capable of 1920x1080 or higher at 24 fps or higher.
BTW the page is a little messed up.

About frame grabbers.
Manufacturers are not working with PCI-E at this moment cause PCI-X 64 bits 133 MHZ is still the fastest interface availabe (under the PCI standard) and will be for the next year.
Athough PCI-E specifications allow for bigger transfers, they are not planned to be implemented in the near future except for GPU.

Ben Syverson · July 19th, 2004, 07:40 PM

Steve et al,

The Eden processors themselves don't do RAID, but motherboards like the mini-itx designs at http://www.mini-itx.com definitely support PCI cards which you can do RAID through. Check out this project on mini-itx.com -- a guy created a RAID of notebook drives and hooked them up to his mini-itx via a PCI ATA controller. Odds are you'll be able to do the same thing with the coming Nano-Itx boards.

Of course, then the problem becomes how to deliver enough power to the board and the drives. I don't know the figures offhand, but while these boards draw way less power than a typical desktop, I think they draw more than most laptops. Then your problem becomes finding a DC-DC converter to regulate the current, and finding a battery that can feed it. Oh, and what are you going to use for an LCD monitor?

The whole setup is likely to be to be pretty large. Like one of those ENG (news) video cameras.

On the other hand, laptops keep getting faster and adding more features. And they have the whole power/battery/LCD issue solved. What we need are Firewire a/b or Gigabit Ethernet cameras.

- ben

July 17th, 2004, 10:58 PM	#832
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	The real point... Les wrote: People willing to play with non standard and somewhat self made hacked together hardware are more likely PC people. Yes, most definitely. I'm talking directly to the vendors: if you wanted to, you could develop an HD system that didn't need to be "hacked together," one that could be a much more mainstream solution. Think of all the FCP jockeys out there begging for a better format than DV or HDV. The filmmaking market is at least as big as the scientific market -- trust me. And a lot of those people are Mac users. I can name 10-20 people off the top of my head who would put down a deposit right now if you could deliver them a camera that pushed HD footage at 24fps through USB2, Firewire or Firewire800. You could sell such a camera for $3000 or $4000 (with $1000 - $2000 worth of parts) that would appeal to thousands of videographers, filmmakers and others for whom HDV is not going to cut it. But CameraLink will not sell units to this crowd, except a few dedicated individuals like Obin. This whole thing is set to explode in the next 18 months, and whoever is out there first stands to gain the most ground. Here's what you need: a 720p camera that can deliver at least 24fps over one (ideally a couple) of the interfaces I mentioned at 8 or 10bit. 12bit is overkill for the most users. 10bit 720p at 24fps is around 26MB/sec, which most 7200rpm drives can keep up with. Maybe do some compression via a FPGA in the camera to get it down to 18 or 20MB/sec -- by the time you have a shippable product (12 months?), you'll have no trouble approving laptops to handle it. Software vendors are screwed -- this market won't like kludgy proprietary tools. We'll build our own or buy better integrated tools. In the next 18-24 months the "industrial" camera websites we all visit will cater to both the scientific and filmmaking communities. Some may only cater to the film community (since it's potentially far more profitable). Be there first. - ben

July 18th, 2004, 12:53 AM	#834
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Physical bayer registration Obin: We all give you props for being our Chuck Yeager. :) Jason + I ran into an interesting problem today. He noticed a "gridlike" effect in the de-bayered results of my filter, when I knew there was no reason for that effect to occur normally. I looked into it, and we found that the green pixels in odd rows were often globally different than the green pixels in even rows. When I analyzed the pixels, I found that the green pixels in a blue row might be (in 8bit): 100, 98, 99, 97, 96, 98 but on the next row, a red row, the green pixels might be: 92, 90, 90, 89, 85, 88 Steve, have you seen this before? The rows were sometimes off by as much as 10 or 12 values, which is a significant deviation. The only logical explanation for this is that the physical bayer filter grid on the sensor is veeeeeeeeery slightly mis-registered. Like it's a little up or down of where it should be. As a result, the blue and red filter areas are covering up the green sensors a little bit, and since blue blocks more light than red, you get a value difference. The red+blue channels come out even because they're being corrupted equally by only the green filters. But the greater the registration error, the lower the accuracy of the color. I verified this hypothesis by analyzing pixels from areas where the R & B are very different, and where they're almost the same -- sure enough, the greater the contrast between R&B, the more "interlacing" artifacts in the G. This isn't Micron-specific, I've seen this on images from the IBIS5 sensor as well. It must be a manufacturing nightmare to register a filter on a 2/3" megapixel + sensor, when your margin of error is around 1 micron -- 1/25400th of an inch! So guys, this is yet another reason to buy sensors with bigger pixels -- not only does their light sensitivity go up, but the bayer registration is almost certainly better. If you have 12 micron pixels as opposed to 6, an error of 1 micron won't be so dramatic. I'll build a slider into linBayer to counteract this effect on the green channel, but using it will necessarily affect the original values of the image, and thus the process will not be totally lossless. - ben

July 18th, 2004, 08:36 AM	#837
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Ben on zippers: The Micron has 4 analog gain amps - R, G1, G2, B. This means odd and even rows have their own gain setting. Either they are set differently or they are not balanced in the original image. For Obin, I would suggest live viewing and zoom way up on a solid color image. That is how we balance columns on the IBIS5. I haven't heard a FG manufacturer talk about PCI-E. PC-104+, yes. Ben on Bayer: I think you hit something very significant when you said that the original pixels are not modified so you can go back at any time and re-do the Bayer. I hadn't thought of that before. It means that if you find an incredibly complex algorithm, you could run it on the final cut of the film if you stay had high bit depths through editing. Smearing: In about 1.5 weeks I'll let you know about the SI-3300 smear - micron says it should be much better. I haven't heard of the Altasens smearing any which way. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

July 19th, 2004, 07:40 PM	#840
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Via-based designs Steve et al, The Eden processors themselves don't do RAID, but motherboards like the mini-itx designs at http://www.mini-itx.com definitely support PCI cards which you can do RAID through. Check out this project on mini-itx.com -- a guy created a RAID of notebook drives and hooked them up to his mini-itx via a PCI ATA controller. Odds are you'll be able to do the same thing with the coming Nano-Itx boards. Of course, then the problem becomes how to deliver enough power to the board and the drives. I don't know the figures offhand, but while these boards draw way less power than a typical desktop, I think they draw more than most laptops. Then your problem becomes finding a DC-DC converter to regulate the current, and finding a battery that can feed it. Oh, and what are you going to use for an LCD monitor? The whole setup is likely to be to be pretty large. Like one of those ENG (news) video cameras. On the other hand, laptops keep getting faster and adding more features. And they have the whole power/battery/LCD issue solved. What we need are Firewire a/b or Gigabit Ethernet cameras. - ben

July 17th, 2004, 05:08 PM	#826
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	when will we have pci E frame grabbers Steve? that is a VERY fast standard Shuttle has a new board that has one pci E slot..news??

July 17th, 2004, 05:26 PM	#827
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	Obin and Steve, So If I add a mechanical shutter at 1/48th to the sensor and run it at 24 fps (not 48 to avoid rolling shutter) I won't have the smearing problem? I ask cause, if this solution gives better quality I'd go this way. If someone is interested DCRAW uses the Variable Gradients Method.

July 17th, 2004, 05:40 PM	#828
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Variable Gradients is an interesting approach, but from what I understand, it's fairly slow. My algorithm could probably work in a realtime pipeline if you optimized it. Processors with SIMD (single instruction multiple data) such as the G4/G5 and the newer Pentiums should have no problem with it. Also, I think it could be put onto a FPGA pretty easily if that ever becomes a reality... Hey, I posted the PC version of my updated linBayer plugin here for you guys to check out. Jason + I have been working on getting all the kinks out, and I think it's to a pretty good point. I was able to eliminate some artifacting in noisy areas in this latest build. David wrote: We don't have any Macs or a development enviroment for a Mac. At least for today all CineForm tools will be PC based. Sounds like you're missing out on a massive chunk of the market (if you're at all interested in this market). Mac users probably make up the majority of people interested in low-cost HD solutions for filmmaking. Just a thought. $10k investment for a possibly massive return...

July 17th, 2004, 08:13 PM	#829
Les Dit Major Player Join Date: Oct 2003 Location: Southern Cal-ee-for-Ni-ya Posts: 608	Might be the opposite. People willing to play with non standard and somewhat self made hacked together hardware are more likely PC people. Ever been to a Computer swap (fair) ? Here in LA it's like two football fields of vendors with all shapes and forms. No Macs, except a couple of old junk ones in a used pile for $5. Maybe people who hate being locked into a single source for the computer also eschew the big name video vendors, like Sony and Matsushita. I like open hardware, myself.

July 17th, 2004, 09:03 PM	#830
Jason Rodriguez Trustee Join Date: Mar 2003 Location: Virginia Beach, VA Posts: 1,095	That's okay, we Mac users replace not having open hardware with open software ;-) I do think it's pretty cool that since OSX/Darwin has hit the market, I've been able to play around with all the goodies of the OSS market while still enjoying "mainstream" apps like Adobe, FCP, etc. I'm sure you can do a lot of similar stuff with cygwin, but I've found many GTK-based stuff, or anything that wants X-windows to be pretty buggy on Windows versus a native X interface like I can get in OSX (special install). Also Ben, I bet if you wrote that linBayer as a CoreImage/CoreVideo plug-in, you'd be seeing real-time results. Of course that won't be released till "Tiger" comes out, but I've seen it do real-time gaussian blurs, etc., and those filters take a lot longer to process on my machine (dual 2Ghz G5) than your linBayer filter does in AE-so I'm assuming that it's less processor intensive than a heavy gaussian, and if that plays back RT in CoreImage/Video, then your filter should be a real-time conversion as well. P.S. BTW, I'm not trying to make this a Mac/PC war, I realize that right now I'll have to purchase a PC anyways if I'm gonna have one of these industrial cameras.

July 17th, 2004, 10:38 PM	#831
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	OMG CoreVideo looks HOT!!! I hope someone does this for windows soon...we are all using $100 agp cards that could do EVERYTHING in realtime even HD ..yet not a program uses them but games...shame...it would be Apple that changes that ;) and I am a windows person

July 17th, 2004, 11:07 PM	#833
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	I was here first ;) looks like windows xp embedded is a good os for the camera

July 18th, 2004, 01:29 AM	#835
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	More cameras: http://www.mikrotron.de/en_cameras_m...n.shtml#mc1301

July 18th, 2004, 03:58 AM	#836
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	After playing with more of the Micron images, the bayer registration seems to be pretty solid. It's the IBIS-5 images that need more help. I posted the updated version with registration correction, and more control over the de-zippering process.

July 18th, 2004, 03:38 PM	#838
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	Steve: I don't see how we can use the 3300 at all as your going to window the chip for 1280x720 and even for 1080p..it's a 1/2 inch chip as-is so making that chip work with a decent FOV...I don't see how unless you can do some type of superpixels/binning ?? OR and this is the BEST if you could resize the native image BEFORE it is saved with a realtime dsp/fpga chip?? if you could resize it would make that chip BETTER then the Altasens chip because it will be MUCH sharper @ the resolution of 1080p or 720p because it's been downsized..this would be VERY good... If that could be done I would take the 3300 over the Altasens

July 19th, 2004, 12:40 AM	#839
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	About SI 3300, Isn't it a 1920x1080 capable Sensor?? Am I missing something? edit:Oh, sorry.I went to SI's page And I read it is a 2048x1536 chip but at 15 fps. Does SI have a chip capable of 1920x1080 or higher at 24 fps or higher. BTW the page is a little messed up. About frame grabbers. Manufacturers are not working with PCI-E at this moment cause PCI-X 64 bits 133 MHZ is still the fastest interface availabe (under the PCI standard) and will be for the next year. Athough PCI-E specifications allow for bigger transfers, they are not planned to be implemented in the near future except for GPU.