4:4:4 10bit single CMOS HD project

[Steve Nordhauser]

Obin on transfer time:
Don't do any processing on video on a network drive - move it locally and then back.

Jim on a control system:
A couple of things to watch. First is cost - PC motherboards are pretty cheap. If you are grabbing camera link, you could embed your own interface - not too complicated. How will you do the viewfinder? Pretty easy with a PC chassis. What about a smaller footprint like PC/104+? I know there are frame grabbers out there for it. Not extendable to 64 bit, as far as I know. There are some other standards for SBCs (single board computers) like this EBX format:

http://www.orbitmicro.com/products/e...370/EM370B.htm

It has PC/104+ so you could attach a frame grabber. There are many - don't get hung up on my first google search. You might find one with a RAID controller. Motherboard and PCI grabber will always be cheaper.

Jim on software filters:
It should be easy to do some kind of plugins in version 1.1 or so of the software. Color balance is just a 3x3 transformation table. I think photoshop lets you build your own filters by editing the table. Hey, why not use the standard photoshop plugins or whatever video editing tool uses plugins.?

[David Newman]

Obin,

Well you already know the problems of 8bit, that is what you are trying to avoid. :)

Regard the 10 or 12bit issue. There is little point for applications to support a native 10 or 12 bit workflows when computers work better on 16bit. Our codec is 10bit yet our workflow is 16bit, this is completely normal. 10 or 12 bit data is simply left shifted to be processed as 16bit. This is then reversed when exporting back to the codec. The extra interim precision is a good thing.

[Obin Olson]

sorta like filling your gas tank 3/4 the way full? so the image file is not bigger then a 10bit EVEN though it's 16bit?
David I sent you 2 cd-roms with some raw data on them, HTH

So we need to have a 16bit compression codec to edit with and a way to edit it in a normal edit system like premiere pro. We could do our color work in 16bit and save the files for editing in 8bit and then master in 8bit onto dvd HDDVD mpeg-2 HD DVCPROHD etc?

here is the card to get for this system

http://www.highpoint-tech.com/USA/rr1820a.htm

upto 1200MB/sec with 8 SATA disk drives and the card is under $200!!

[Jim Lafferty]

Obin and Steve -- thanks for the responses.

I know I might be getting ahead of the game a bit, Obin, but I'm trying to think ahead conceptually so we can best accomodate certain design ideas from the outset. I find this helps when trying to avoid potential future pitfalls.

That said, I've a friend who's one of the PHP codifiers (not just a coder -- he helps standardize new PHP commands). He's got a lot of coding knowledge and is doing render/compression stuff for 3d environments at the moment. I'm hoping to persuade him that this project might be worth his time.

I've also taken the liberty of setting up a quick-n-dirty site of still images from the footage you've provided, Obin. Here it is.

I hope you don't mind -- I'm just trying to get the word out and I find the images are worth many words :D

Also, it helps for people who don't have WM9HD installed.

The site will be updated with condensed info and links to footage, images, etc. as I have time and as the results here merit.

Thanks again for the work you're all doing!

- jim

[Les Dit]

I've been dealing with the bits problem for about 10 years.

For now, just to save disk space, I recommend using the Cineon ( fido ) format, it puts three 10 bit rgb code values into 32 bits ( 4 bytes ), so you only waste 2 bits per pixel.
That's for rgb demosaiked frames.
You could pack the bits in a similar fashion for the raw bayer image, and decode it later. It's all in the name of saving disk space.

Like I said before, LZ or RLE won't help much with > 8 bits.
But if you have to, the LZ Tiffs would crush out the wasted empty bits to give similar files sizes, but with more cpu to get there.

-Les

[Rob Lohman]

Okay, let's all slowdown a bit. Obin: I understand your frustration,
but this is what a lot of people have predicted and we know
would happen.

That's why we are trying to design a system that works like a
camera and not something with 20 steps to go through. As I
explained earlier, this is NOT going to happen overnight. Although
Rob S. and myself are working on details and testing things, we
both have jobs and Rob S. is still waiting on his camera.

Development of software, testing algorithms, optimizing, testing
systems and generating a workable platform takes time. Lots of
time. Where you expecting to be up and running in a couple of
weeks? That's just unrealistic.

David: is 10 or 12 bits always left justified? Is there a reason for
it being left instead of right?

Jim: the main problems is bandwidth / datarate and the system
to put inbetween the camera and the harddisk (or a set of
harddisks). Two programmers (Rob Scott and myself) are already
on the project so, yes, a middle system is being worked out but
don't expect the "magic button" (as Obin calls it) next week or so.

Les: Rob and myself pretty much agreed on doing no Bayer
conversion in the "camera" section. That takes way too much
time and resources to do good and we don't have that (we
must process a very large data stream).

So we are going to write the RAW stream coming from the camera
to disk in at least a packed form. This means that 10 bits per pixel
turn into 40 bits/5 bytes per 4 pixels instead of 8 bytes.

Storing it in Bayer form (10 bit) gives us a data reduction (not a
true reduction since this is what is coming from the camera, but
a reduction in comparison to full demosaiked RGB data) of 3:1.
This reduces bandwidth 3 times or 77%. Then we at least store
it packed which saves us 37.5%.

So for every 24 bytes (not bits) of RGB data we only store 5
which is a 4.8:1 compression or a reduction of 79.2%. This is
without loosing any quality (except for the Bayer filter, but that
you can't get without on single chip system either except for
a Foveon chip)

Since the camera is sending us raw Bayer at 16 bits this gives
us a 37.5% reduction or a 1.6:1 compression.

Rob S. and myself (and everyone is invited, ofcourse) are looking
into ways to futher LOSSLESS compress this. Rob has already
seen some codecs. I'm a bit worried about speed and we should
keep this as fast as possible (speed over compression) to deal
with the bandwidth (see below).

At the moment we are not looking at any compliant file because
I doubt there would be one for Bayer format (except RAW camera
files. Canon has CRW I think).

After it is recorded the signal must be dealt with on a computer:

1) convert from Bayer to full RGB with the best (selectable) algortihm

2) do things like white balancing / color correction (can only be done with the full RGB data)

3) write it out to a standard format / codec (perhaps with smaller resolution proxies [both in resolution and bit depth] to ease the editing)

Now I hear a lot of people think I thought the camera would do
white balance and color presets and whatnot. Don't count on it.
We probably don't have the processing power. Plugins for the
second phase of the capture process should be no problem.

The problem with that is, is that the data must be converted into
full RGB first before we can do anything with it. This also increases
our storage three-fold. What we might do is do some lower
resolution stuff for the viewfinder / monitor.

Instead of full bayer you simply make 1 pixel out of every macro
block that contains 4 pixels (2 horizontal, 2 vertical). This gives
you a 50% reduction in either resolution and a very fast de-bayer
algorithm. So a 1280x720 image becomes 640x360 which is more
easily displayed on a standard monitor as well. We might have
time to do a bit of color-shifting (white balancing) on that and
it might be possible to save such presets with the stream so the
second stage on a full blown computer can take that as the
default setting.

So for now it will definitely be a two stage process. Capture and
store and then the second phase transform and transcode to
a different format.

The list of the most problematic things in this project are (in my
opinion):

1. working out a system to go inbetween the camera and storage (full PC, FPGA, embedded chips etc.)

2. working out a storage system that can keep up (or compression)

3. getting a viewfinder / monitor out

Now to give everyone a low down on some data rates (again):

1280 x 720 x 24 fps x 8 bit, full RGB = 63.28 MB/s (126,56 MB/s)
1280 x 720 x 24 fps x 10 bit, 16-bit full RGB = 126.56 MB/s (253.12 MB/s)

1280 x 720 x 24 fps x 8 bit, Bayer = 21.09 MB/s (42.19 MB/s)
1280 x 720 x 24 fps x 10 bit, 16-bit Bayer = 42.19 MB/s (84.38 MB/s)
1280 x 720 x 24 fps x 10 bit, packed Bayer = 26.37 MB/s (52.73 MB/s)

1280 x 720 x 24 fps x 12 bit, packed Bayer = 31.64 MB /s (63.28 MB/s)

The numbers between the () are for 48 fps. I've added 12 bit on
the last row to see where it might be headed. So the most optimal
size and very easy to implement (10 bit, packed Bayer) is the one
that will definitely be there. Whether or not we can support any
futher compression depends on a lot of things (licenses, speed,
resources etc.).

Steve: can you explain to me in a little sentence what PC/104 is?
Is it like a form of mini-PCI or PCMCIA?

[Steve Nordhauser]

Rob L: PC/104 plus. One sentence- easy:
http://www.pc104.org/technology/pc104_tech.html

I'll elaborate. PC/104+ is an small stackable card format designed for industrial uses that mimics PCI-32 in signals. Instead of card edge connections (we've all had to reseat boards in computers) it uses connectors like hard drives use and the boards screw together. Since it mimics the standard PCI, designing a new board is usually just a relayout. That means there are a lot of boards out there. Price is definitely higher that PCI (volume of sales are lower), but possible for the integrated camera idea (CPU + LCD controller + disk controller) but the CPUs tend to lean towards lower speed, lower power. EBX is worth watching (same site). It is a somewhat larger embedded format that uses PC/104 plus as an expansion board. If I were designing an embedded PC product, this is where I would be looking. It could also be a migration path for what is being developed without any software changes.

By the way, Rob L, nice laying out the project for everyone. I think you spelled out all the major first pass design decisions. A 640x360 viewfinder should be able to be done with a cheap, small LCD screen.

A non-programmer who would like to assist can look for a mini-pc box, like the shuttle with a low noise power supply (or the PS from another source). Someone posted this:
http://www.logisysus.com/

I was told that it is better if you have a separate OS drive from the video. So, ideally, the box should be able to handle a small drive (laptop?) for the OS and two 3.5" for a RAID. Maybe removable. Definitely gigE. For people doing larger work (as Obin dicovered), they may need to move it faster. Or we be patient - I think 10gigE will be dropping quickly in price.

[Valeriu Campan]

Steve,
I am coming back to the CMOS chip saga. I found something that looks almost incredible. I would like to know your opinion or anybody else of course.
The Lupa 1300 from FillFactory, probably the same manufacturer that supplies the Sumix camera:
http://www.fillfactory.com/htm/produ...0/lupa1300.htm.
It's size is only 30% smaller than full 35mm frame, has 62db S/N ratio, is capable of 30fps. I think that for a start (1.3Mpixels Bayer), is not a bad choice.
Any thoughts?

[Obin Olson]

looks like a good chip..Steve?

I have a good idea, lets make the mcroatx computer run in RAM so when you turn it on you don't have to wait for windows to load an load and load...yes??

[Steve Nordhauser]

Fill Factory, the company that brought us the much maligned IBIS-5. This is the direct competition to the Micron MT9M413 http://www.micron.com/products/imagi...s/MT9M413.html
Noise spec is somewhat better, but nothing like the Micron 1.3Mpix rolling shutter or the Altasens. Simultaneous expose and readout - that is very good. 16 analog taps - very bad. This would be a very expensive camera (not a cheap sensor either).

I guess I haven't heard a consensus that you are willing to put up with noise and high cost (especially for 720p) to gain a global shutter and large pixels? Would you pay $4-5K for this camera? The Altasens will be in the same ballpark with 1920x1080 60fps true 12 bit.

[Valeriu Campan]

4-5k is tooooo much for a prototype. Altasens with those specs sounds more promising. What is the size of the sensor? Still 2/3 or larger? Don't forget that the big boys are looking at full frame 35mm (Dalsa, Panavision, Arri). Also many from this list are aiming for a similar outcome.
The look given by the combination of FOV and DOF are still an important issue for narative long form projects.

[Rob Scott]

Rob L - Thanks for excellent summary! Do you mind if I adapt it for my wiki on the project?

Obin wrote:
I have a good idea, lets make the mcroatx computer run in RAM

It is possible to put a CompactFlash drive in a PC that looks like a standard IDE drive ... but that would only help if Windows can be installed in 1 GB :-)

Valeriu wrote:
What is the size of the sensor?

The AltaSens is 2/3" I believe.

[Obin Olson]

I was talking a RAM drive..can't you install windows into ram?

http://www.cenatek.com/product_rocketdrive.cfm

EDIT:
here we go:
http://www.bitmicro.com/products_edisk_25_ide.php

[Eliot Mack]

<<<-- Originally posted by Rob Scott :
It is possible to put a CompactFlash drive in a PC that looks like a standard IDE drive ... but that would only help if Windows can be installed in 1 GB :-)
The AltaSens is 2/3" I believe. -->>>

Is the actual target platform for the camera Linux? If this is the case it may be much easier to get it onto a CompactFlash drive. Not sure what would have to be stripped from Linux, though. I may check this out a bit as I'd like to avoid a camera boot cycle.

Eliot

[Rob Scott]

I was talking a RAM drive..can't you install windows into ram?

I don't think so. The biggest problem is that RAM disappears when the power is off, so when you power on, the PC has to create the RAM drive from scratch. I suppose it's possible that there is a boot loader that will pull Windows off the hard drive and copy it onto a RAM drive first, but that doesn't seem much quicker than just booting.

Edit ...
I looked at the Cenatek device, and (unless I'm mistaken) it requires power all the time. When the power goes away, so does the contents of the RAM.

Is the actual target platform for the camera Linux?

That's a long-range goal. I'm not sure if/when that will happen, because I don't have much experience with development under Linux.

There are a number of "embedded" distributions of Linux, some small enough to fit on a single floppy. Linux on a CompactFlash card is definitely doable.