Affordable lossless HD storage
 

To cope with the data rate of HDTV and digital cinematography Sony has brought out a new tape format HDCAM-SR and two VRTs that use it to record 1920x1080 60fps 4:4:4 with a lossless MPEG4-Studio compression.

Now these are certainly good machines, but at $88k for the non-portable VTR, and $100/hr tape stock, well I wolud be suprised if many members of this list splashed out on them 8-o

But it would not suprise me if lots of people wanted to record 1920x1080 25fps 4:4:4 if they could find a more affordable way. So I did some homework...

Direct to disk at this data rate probably means a 10,000rpm scsi RAID on a CameraLink framegrabber, eg the IO industries CL160 32bit PCI, or its 64bit big brother.

Current big scsi drives aren't cheap eg Hitachi Ultrastar 10k300 gives 300GB for $700, ie $2.33/GB. Now 600GB will give you about an hour of Bayer RAW from a single chip. A 3 chip sytem will treble your captured colour resolution and total pixel count, so you would need 6x300GB => $4200/hr! Which makes even 35mm cine look like a bargin....

ATA drives are slower, cheaper, bigger. The Hitachi Deskstar 7h500 gives 500GB for $600, ie $1.20/GB. You could unload your scsi drives onto ATA drives for storage, live playback wouldn't be possible, and it would interupt shooting for a few hours. It would be wise to start emptying the scsi's whenever you weren't shooting.

There are some good lossless compression codecs to choose from (see Wikipedia's page). The Huffyuv, MUS and LCL codecs are freesoftware, and can be used with the freesoftware capture app Virtualdub on a range of platforms including WinXP. Being free you have the source code and the option of tweaking it if you have that skill. From 4:2:2 they will give a bit more than 2:1 lossless compression, from 4:4:4 they will give about 4:1 compression. Huffyuv is faster, LCL is more compressed typically. If 4:1 compression is achieved then a 500GB ATA drive will hold 3hrs20min Bayer RAW, or 1hr6mins of 3chip YUV. (NB typically better compression is achieved by these codecs in YUV colourspace, than in RGB colourspace.) That would give $90/hr for single chip, or $273/hr for true 4:4:4 from a 3chip system.

Thats about the cost of 35mm, but the media is reusble (important for wildlife work with very high wastage rates), and it's non-linear. You will probably have to uncompress it to the scsi drives to actually work with it on an NLE though. It could be quite compact and portable, suitable for walkabout filming if you needed to.

Hard drives are fragile, and a typical 1hr documentary might need 15-30hrs of takes especially if you want a verite style.

Tape is a better bet for mass data storage, and long term backup. Consumer DAT tapes are out, the data rate is far too low. Rackmountable enterprise drives are more suitable.

IBM makes the Ultrium range of scsi Ultra-160 interface tape drives. The Ultrium 3580-L33 has a transfer rate of 80GB/sec, and takes the Ultrium 3 data cartridge which holds 400GB and costs $116, ie $0.29/GB or $130/hr for 3chip system. Which is 30% more per hr than HDCAM-SR. The tape drive costs $5999, wich is the same as 21hrs on ATA drives, or 1.5hrs on scsi drives. These drives work on Windows 2000, 2003, Linux, and *nix.

Realtime tape recording of 1920x1080 25fps 4:4:4 would require at least four drives and a controller to spread the data over them. That would workout about $30k. Which is more than I would consider, but less that half the Sony SRW1.

I would reckon realtime lossless HD is out, but six scsi drives and one tape drive is a reasonable approach. It might not be necessary to use the biggest scsi drives available, if you took every chance to transfer to tape.
You would need to shoot about 2,600 hours to pay for a HDCAM-SR VTR instead.

Storage grows even faster than processor speed, so all this info will date very fast. The time today is yesterday.....

Enjoy,

Nick Hockings

Portable 30fps to tape
 

Bit more searching revealed a faster tape drive:

HP offers the Ultrium 960 with a sustained write speed of 80Mbit/s (160Mbit/s at 2:1 compression). Works with certain windows and most *nix-es.

The internal version of it weighs 3.6kg, and measures 8.3x14.6x20.3cm.

If that compression can be applied to camera link data being written to scsi, then only three drives (one per chip) would be needed.

Current asking price is around £2.7-2.2k thats around $5k.

A portable server is certainly feasible, and applicable to ENG, documentaries and Indy productions. If you want to run, then you could use a single CCD bayer chip, and one drive.

NB I'm taking my data rates from the Imperx 2M30DH-L, which uses a true 1920x1080 progressive scan CCD at 8,10, or 12 bits of resoloution.

Thanks for that, we have had "experts" around here telling us that average lossless doesn't exist much over 2:1, which sounded too low. But that would be because of the limitation of compressing 4:2:2 data, where as 4:4:4 gives more parallelism in the colour channels.

I have been following your efforts in the back ground here, and have also being doing some research on lossless compression at the moment myself. I have been coming up with new compression methods over many years to implement into my OS project (when the time comes) and started reading up last week on the industry compression methods, as I am actually thinking about implementing a lossless codec now. I've written up some stuff on my web based compression research, for a post on the Digital Cinema Camera Technical discussion thread.

I can tell you some things now, if you look for the NewsNet faq on compression you will find a link to lossless compression tests that have some interesting figures for pictures from file archivers, there was also another one I posted on the thread but can't remember it's name. Though not terribly quick it is still worth a look at, as increasing computer power is always making it quicker. I have been looking up information for group 4 fax and it's JBIG replacement, because I had heard in the 80's of very high lossless compression ratios, and from my search I have heard figures upto 40:1. But don't get too excited though, it might be able to compress that for text pages and the like, but it is still one of the best methods for 4:4:4 video compression (out doing many standards for 6 bit images). Processing requirements are also less I think. They do this by applying it separately to each channel. Over 6 bits gets to be a problem (for all the routines) due to noise and details in these channels, but I say better lighting and cameras should get rid of that noise.

http://www.faqs.org/faqs/compression-faq/part1/

http://www.faqs.org/faqs/compression...rt2/index.html

Hardware list for compression
http://www.faqs.org/faqs/compression-faq/part3/

Most of the compression stuff looks fairly primitive and wasteful. I understand that Mpeg4/H264, fractals, and DVI has some modeling functions that might turn out to be similar to mine, but I am yet to have a look. When you get to the level of modeling functioning you see high compression, and even across frames. For instance during the 80's they could achieve 6000:1 for tele-conferencing compression of a persons head, simply by modeling the faces features and transmitting the changes in the model for the lips (I think). But the problem is that this sort of modeling takes up too much processing resources, but instead they are already wasting too much on simple compression (wavelets/Jpeg2000 seem to be leaders in quality and processing requirements (JBIG seems to be remotely related). With things like DSP's, Cell processors (Playstation 3 and all comers) GPU programming, and newer FPGA's (go to www.opencores.org/browse.cgi/by_category to see) will be within reach sooner.

(Other threads of information)

Have you seen the threads for the digital cinema camera projects, they have covered much of the ground you are seeking to cover already, with lots of links to lossless compressors (BBC has one) storage and other technologies have been posted their.

http://www.dvinfo.net/conf/showthread.php?t=28781

http://www.dvinfo.net/conf/showthrea...threadid=25296

http://www.dvinfo.net/conf/showthrea...threadid=25705

http://www.dvinfo.net/conf/showthrea...threadid=25808

http://www.obscuracam.com/

You will find links to many of the newer threads in these as well (though in the last 3 months I haven't been posting the newest ones).


(Tapes)

I posted some stuff about server backup tapes on the Digital cinema threads, I think some much higher data rates then you mentioned. The costs of drives (500GB 50MB/s sustained each, smaller ones upto 70MB/s+) is much lower than tape, capturing is the costly/hard part where it becomes very tricky is memory buffering, packing pixels and finding the resources to compress to save disk space. To keep things secure you might be better off backing up the disks in between takes, or mirroring it live, and keep the other disk in a separate stable secure position. Some of these disks will take great shocks when their heads are parked, so setting the the head to park instantly when not in use would help.

Well, have fun on your project when it gets here.


Wayne.

Thanks for the links Wayne.

My key aim is to have something rugged & portable with about 30hrs of data. That way I could use it for international and remote area documentaries, as I have with my XL2.

At the moment a single chip Bayer system would be, especial if I used a single Ultrastar 15k147 with sustained write speed of >90MBs. The trouble is Bayer is 1:2:1 rgb. I'd really like to make a 3CCD system.

I'm still waiting for replies from Optec (trichroic prism makers) and Imperx.

I agree, and I would to keep track of your three chip efforts, because when I mentioned the possibilities I was shot down in flames for suggesting it. But the people weren't being real, you can even make it your self (google for DIY clean room, and containers). It might not be for everybody (hard work and costs) but then it doesn't have to be if you plan on doing it as a business.

If you are thinking of DIY then you might like to consider pixel shift methods. A 720p with 1/3rd pixel shift 4:4:4, yields over 8MP (where each pixel is shifted 1/3rd to the last to produce detail at 3 times the resolution a side, or another words 9 times the resolution you are recording) enough for the new (see blog at HDforindies.com) digital cinema spec. Recording 108025p 8bit works out approx 16MB/s. The method should not be that precise, but because images are somewhat predictable between channels and edges (which Bayer depends upon) the image should be enhanceable through software processing. I have written in the threads concerning the idea (and written threads specifically about it). I am mainly interested in it for a poor mans camera (I think the Pana GS120 uses it to get 1.2MP stills) but not as good as a plain 3 chip.

Anyway, have fun.

Up sampling by pixel shift or Bayer conversion (as done in most DSLRs) doesn't increase the information in the image. It just presents it a a more aesthetic form than simple enlargement. People don't like to see the pixels, so blurring them looks nicer...

A 1920x1080 3ccd image contains as much data as a 6MP bayer image (eg a 2004 model Nikkon D70), only it has a sustained frame rate of 30fps or more. Infact the 3ccd image has as much red and blue data as a Canon D20 8mega-pixel bayer image.


A cheap 3ccd arrangement could be made with cube beam splitters as used in digital projectors. My concern would be the blurring caused by where the joins cross in the middle. Cube beam splitter are true trichroic prisms, and can be bougth in many different sizes from both American and Chinese suppliers.

Optec appear to be the premier trichroic prism makers for cameras. The make a wide variety for the machine vision market and I suspect they supply many of the camcorder makers.

Optec also make custom di,tri,tetra, and penta chroic cameras with interchangeable chips,filters, and lens mounts. It is a variant of their semi-standard 2kx2k camera that I am waiting to hear about. Such a head would be upgradeable with new (faster) chips when such become available.
A 35mm version might be even more appealing.


The rest of my project is sorted out down to a range af affordable(ish) options. The 3 PCI =>PCMCIA adapter is no longer available. A run-around version is currently not possible. I believe a single ccd bayer would be no better than the available HDV cameras. When 3xGBit Ethernet cameras can be plugged to to a tablet PC..... well by then there will be min-HDCAM-SR or something like it.

A portable version on a leash to a rugged server is feasible. I would use Promise SuperTrak EX8300 (8x SATA300 RAID PCI-X 64/133) 64MB
€ 365,55 (Ex VAT) from www.webconnexxion.com
with >6x Hitachi Deskstar 7k500 half TB drives,
two IO-systems CLCF 68 pin PCI-x frame grabbers
a sound card with XLRs, a 64 bit server mother board and one or more 64 bit CPUs, all in a rugged box from www.acmeportable.com.

The cost would probaly exceed an XL-H1 plus HDCAM vtr, and be less portable. BUT it would have higher res, and be component upgradeable. It would be my full HD editing suite up whatever mountain I could lug it.

It would be good for Indy, and the best possible for wildelife and any doco where you didn't have run or hide. And then digital HDTV will broadcast it at 15MB/s .........

PS The rugged server could equally take a Black Magic HD-SDI card and out do any HD-CAM-SR vtr, or take any high speed machine vision camera.... 500fps 1MP cameras are about the same price from Imperx or Basler.

Yes, pixel shift would give you less accuracy in colour and the finest detail than 3 chip, it is really only a compromise to get to a software filtered resolution under low bandwidth.

Have a look at this 3M vikuiti one:
http://products3.3m.com/catalog/us/e...er/output_html

Dolch also had good ones, but more expensive.