Instead of Uncompressed what about Lossless? at DVinfo.net

Deke Kincaid · November 7th, 2009, 05:28 PM

I'm just curious if some type of lossless compression like BitJazz, Lagarith, Huff YUV, FFV1, MSU or even some type of lossless compressed image sequence(exr/tiff with lzw/zip compression) has ever been thought for the "uncompressed" option on the Flash XDR? It would keeps the data rate down when writing to Compactflash without having to resort to using striped cards.

I don't even know if this can be done in the hardware so the question may be moot. I'm just throwing stuff out there. With Kona cards for example I use Sheervideo Bitjazz instead of uncompressed since it gets 2.5x compression so I get more streams with my raid with zero image loss.

BitJazz: Products: SheerVideo

-deke

Mark Job · November 7th, 2009, 06:08 PM

Hi Friends:
A few years back Matrox published a rather famous and definitive early work on lossless conmpression circa 1998. At the time Matrox's Digi-Suite was paralelling DPS's Perception & Perception RT real time dual stream editor as the only affordable PC based realtime hardware accelerated NLE boardset and proprietary software bundle. These systems were based on NT 4.0 with service pack 3 or higher. I think you had to have 128 MB of RAM and a Intel Pentium II 233 Mhz CPU.

Matrox DigiSuite LX
http://www.avid.com/resources/whitepapers/DNxHD.pdf

I'm looking for that darn Matrox whte paper on the subject. I'll post it here in this thread if I can find my archived copy of it. Avid's DNxHD codec is designed as a mathmatically lossless codec. I wish the XDR recorded in DNxHD as *an additional available codec* along with the excellent Sony encoding engine used. BTW, it's free to use and Avid will let you use it. They have an SDK for it.

Deke Kincaid · November 7th, 2009, 06:25 PM

Quote:

Originally Posted by Mark Job

Avid's DNxHD codec is designed as a mathmatically lossless codec. I wish the XDR recorded in DNxHD as *an additional available codec* along with the excellent Sony encoding engine used. BTW, it's free to use and Avid will let you use it. They have an SDK for it.

Unfortunately DNxHD is not lossless, it is "perceptually lossless" which is not the same thing. Very similar to Prores. With a little DI & compositing, I can easily break both Prores and DNxHD over uncompressed.

The trouble I have with both DNxHD and Prores is they are limited to certain hardware and operating systems and they aren't really lossless. The codecs I listed above are not hardware/software/os dependent and can be used by anyone (like mpeg). Even better with Bitjazz, which is why I brought it up is all your editing software can read it native, capture with it, works in both avi and quicktime mov, the price isn't very high and the read only codec is free (write codec is only $150 and goes down in price when you buy more then one).

Mark Job · November 7th, 2009, 06:51 PM

Hi Deke:
Matrox has one which is mathematically lossless but it may also be proprietary.

David Cherniack · November 7th, 2009, 10:24 PM

Mark,I'm not sure which Matrox codec you're referring to. I'm an Axio user. Matrox's codecs that I know are rendering codecs not acquisition codecs: mpeg iFrame (that's adjustable bit rate) and uncompressed (8 and 10 bit).

David
AllinOneFilms.com

Marcello Mazzilli · November 8th, 2009, 03:46 AM

I am trying Sheer but my need is not capturing because I capture directly with the camera. It could be usefull afterwards (archiving or switching from MAC to PC) but can't get it to work properly.

I render my XDCAM MOV file in Sheer 10 bit. Works but..

1) Can't get the presets in Premiere so (I think is for this) it goes slower than my original MOV
2) There is no batch feature. Would be usefull to convert all shot footage.

Maybe I am missing something

Mark Job · November 8th, 2009, 06:58 AM

Quote:

Originally Posted by David Cherniack

Mark,I'm not sure which Matrox codec you're referring to. I'm an Axio user. Matrox's codecs that I know are rendering codecs not acquisition codecs: mpeg iFrame (that's adjustable bit rate) and uncompressed (8 and 10 bit).

David
AllinOneFilms.com

Hi David: Yeah, I'm not either now. (??) This goes back to the early Matrox Digi-Suite and their (Matrox) claim that their compression was mathematically lossless. There was quite a debate at the time over this claim I remember.

Perrone Ford · November 8th, 2009, 08:21 AM

There are numerous lossless codecs out there. You can use lagarith (my personal fav), huffyuv, lossless jpeg or jpeg2k, DPX, EXR, Targa, etc. Just depends on the need and application.

Avid's DNxHD at 10-bit is usually good enough for my video work though. But I don't do VFX.

Deke Kincaid · November 8th, 2009, 01:00 PM

I think some of you guys are missing my original point. It was asking if the Flash XDR could do some type of lossless compression in hardware so we could use currently existing cards without having to resort to striping multiple cards together to get the speed.

Uncompressed 422 for 1080p is beyond what even the Sandisk Extreme Pro does:

8 bit @ 1920 x 1080 @ 24fps = 95 MB per/sec
10 bit @ 1920 x 1080 @ 24fps = 127 MB per/sec

8 bit @ 1920 x 1080 @ 25fps = 99 MB per/sec
10 bit @ 1920 x 1080 @ 25fps = 132 MB per/sec

8 bit @ 1920 x 1080 @ 29.97fps = 119 MB per/sec
10 bit @ 1920 x 1080 @ 29.97fps = 158 MB per/sec

Mark Job · November 8th, 2009, 06:00 PM

Hey Deke:
I would like to see the Flash XDR have the choice to stripe 2 or 4 cards together. 2 x 90 MBps sandisk Extreme cards should give you enough sustained throughput to meet the specifications you listed. 4 x 90 MBps sandisk Extreme cards would certainly give you the sustained throughput to meet your stated criteria. I say Raid card number choice makes it a litlle more economical for me.

Deke Kincaid · November 8th, 2009, 08:27 PM

Right.... but how do you then remove the cf cards from the flash xdr read the stripe on your windows/mac box so it will recognize it properly? I don't even know of any usb devices that let you do this. It just sounds very prone to screw ups when you in the field recording and either you mix up the pairs/quads or misplace a card.

The only solutions I've seen are very clunky. One is a pci-express card that goes inside your machine that fits 2-4 cards. The other is a SSD box with a esata port on the back. Especially for in the field, the esata enclosure seems like option that will lead to the least amount of problems (wish it had 4 slots). The last thing I want to do is have to carry a full size pc in the field and then have to open up the inside of it just to swap out CF cards.

Card Reader
Internal Micro SATA UDD from Addonics
CR2T - Compact Flash CF to 2.5? SATA Spanning / Mirroring Enclosure

Do you know of any quad compact flash usb readers because that would be cool?

Daniel Symmes · November 8th, 2009, 08:34 PM

In this case, you would have to play out of the XDR via the HDSDI into a capture.

Come on, we can't have EVERYTHING (in spite of CD's efforts).

Perrone Ford · November 8th, 2009, 09:10 PM

No, we sure can't have everything.

I am going to try to see if I can finagle my way into a Nanoflash next fiscal or if not, the year after that. My requests for these units have remained constant, and CD has hit on nearly all of them.

The only one's I'd still really like to see are:

1. A Wavelet codec.
2. 10-bit log recording
3. The ability to mirror the slots on the Nanoflash

The first would increase storage efficiency. The second would allow the units to play on even footing with Wafian (sans the monitoring capability). The third would alleviate MANY fears for those recording live events.

Frankly, I see this unit as becoming an industry standard, if not as a primary recorder, then as an instant field backup for high-end productions. The guys on the steadicams, can't tie into a Codex or S.Two. This gives them a way to get high quality footage in a tiny package. Put the unit in a backpack or on a belt clip, and go to town.

Best product of 2009 in my opinion.

Mark Job · November 8th, 2009, 09:13 PM

Quote:

Originally Posted by Deke Kincaid

Right.... but how do you then remove the cf cards from the flash xdr read the stripe on your windows/mac box so it will recognize it properly? I don't even know of any usb devices that let you do this. It just sounds very prone to screw ups when you in the field recording and either you mix up the pairs/quads or misplace a card.

The only solutions I've seen are very clunky. One is a pci-express card that goes inside your machine that fits 2-4 cards. The other is a SSD box with a esata port on the back. Especially for in the field, the esata enclosure seems like option that will lead to the least amount of problems (wish it had 4 slots). The last thing I want to do is have to carry a full size pc in the field and then have to open up the inside of it just to swap out CF cards.

Card Reader
Internal Micro SATA UDD from Addonics
CR2T - Compact Flash CF to 2.5? SATA Spanning / Mirroring Enclosure

Do you know of any quad compact flash usb readers because that would be cool?

...Hey Deke: You do not remove the CF cards from the XDR. Convergent Design turns on the FireWire port and you connect the XDR to your PC via that interface and you transfer the uncompressed 10 bit files in this manner. You could also just play out the uncompressed video via the HD-SDI port. I am willing to accept long transfer times in the field to be able to shoot uncompressed 10 bit for digital cinema origination. 128 GB CF cards are on the way. Ooops ! I wrote the un word again ! ;-)

Rafael Amador · November 8th, 2009, 09:37 PM

Quote:

Originally Posted by Perrone Ford

No, we sure can't have everything.

The only one's I'd still really like to see are:

1. A Wavelet codec.
2. 10-bit log recording
3. The ability to mirror the slots on the Nanoflash

Best product of 2009 in my opinion.

I don't know about the XDR, but the NANO, as have been stated, have an SONY MPEG-2 processor. It makes MPEG-2.
rafael

November 7th, 2009, 05:28 PM	#1
Deke Kincaid Regular Crew Join Date: Jul 2007 Location: Los Angeles, CA Posts: 26	Instead of Uncompressed what about Lossless? I'm just curious if some type of lossless compression like BitJazz, Lagarith, Huff YUV, FFV1, MSU or even some type of lossless compressed image sequence(exr/tiff with lzw/zip compression) has ever been thought for the "uncompressed" option on the Flash XDR? It would keeps the data rate down when writing to Compactflash without having to resort to using striped cards. I don't even know if this can be done in the hardware so the question may be moot. I'm just throwing stuff out there. With Kona cards for example I use Sheervideo Bitjazz instead of uncompressed since it gets 2.5x compression so I get more streams with my raid with zero image loss. BitJazz: Products: SheerVideo -deke

November 7th, 2009, 06:08 PM	#2
Mark Job Trustee Join Date: Sep 2006 Location: Montreal, Quebec, Canada Posts: 1,138	Matrox's Famous White paper on this subject. Hi Friends: A few years back Matrox published a rather famous and definitive early work on lossless conmpression circa 1998. At the time Matrox's Digi-Suite was paralelling DPS's Perception & Perception RT real time dual stream editor as the only affordable PC based realtime hardware accelerated NLE boardset and proprietary software bundle. These systems were based on NT 4.0 with service pack 3 or higher. I think you had to have 128 MB of RAM and a Intel Pentium II 233 Mhz CPU. Matrox DigiSuite LX http://www.avid.com/resources/whitepapers/DNxHD.pdf I'm looking for that darn Matrox whte paper on the subject. I'll post it here in this thread if I can find my archived copy of it. Avid's DNxHD codec is designed as a mathmatically lossless codec. I wish the XDR recorded in DNxHD as an additional available codec along with the excellent Sony encoding engine used. BTW, it's free to use and Avid will let you use it. They have an SDK for it.

November 8th, 2009, 08:21 AM	#8
Perrone Ford Inner Circle Join Date: Nov 2006 Location: Tallahassee, FL Posts: 4,100	There are numerous lossless codecs out there. You can use lagarith (my personal fav), huffyuv, lossless jpeg or jpeg2k, DPX, EXR, Targa, etc. Just depends on the need and application. Avid's DNxHD at 10-bit is usually good enough for my video work though. But I don't do VFX. __________________ DVX100, PMW-EX1, Canon 550D, FigRig, Dell Octocore, Avid MC4/5, MB Looks, RedCineX, Matrox MX02 mini, GTech RAID, Edirol R-4, Senn. G2 Evo, Countryman, Moles and Lowels.

November 8th, 2009, 08:34 PM	#12
Daniel Symmes Regular Crew Join Date: Jun 2007 Location: Los Angeles, CA U.S.A. Posts: 146	In this case, you would have to play out of the XDR via the HDSDI into a capture. Come on, we can't have EVERYTHING (in spite of CD's efforts). __________________ nanoFlash Dealer \ nanoFlash 3D, AJA Kona 3G, DeckLink HD Extreme 3D +, Nuke 3D film/video production since 1972

November 8th, 2009, 09:10 PM	#13
Perrone Ford Inner Circle Join Date: Nov 2006 Location: Tallahassee, FL Posts: 4,100	No, we sure can't have everything. I am going to try to see if I can finagle my way into a Nanoflash next fiscal or if not, the year after that. My requests for these units have remained constant, and CD has hit on nearly all of them. The only one's I'd still really like to see are: 1. A Wavelet codec. 2. 10-bit log recording 3. The ability to mirror the slots on the Nanoflash The first would increase storage efficiency. The second would allow the units to play on even footing with Wafian (sans the monitoring capability). The third would alleviate MANY fears for those recording live events. Frankly, I see this unit as becoming an industry standard, if not as a primary recorder, then as an instant field backup for high-end productions. The guys on the steadicams, can't tie into a Codex or S.Two. This gives them a way to get high quality footage in a tiny package. Put the unit in a backpack or on a belt clip, and go to town. Best product of 2009 in my opinion. __________________ DVX100, PMW-EX1, Canon 550D, FigRig, Dell Octocore, Avid MC4/5, MB Looks, RedCineX, Matrox MX02 mini, GTech RAID, Edirol R-4, Senn. G2 Evo, Countryman, Moles and Lowels.

November 7th, 2009, 06:51 PM	#4
Mark Job Trustee Join Date: Sep 2006 Location: Montreal, Quebec, Canada Posts: 1,138	Hi Deke: Matrox has one which is mathematically lossless but it may also be proprietary.

November 7th, 2009, 10:24 PM	#5
David Cherniack Regular Crew Join Date: Sep 2003 Location: Toronto Posts: 186	Mark,I'm not sure which Matrox codec you're referring to. I'm an Axio user. Matrox's codecs that I know are rendering codecs not acquisition codecs: mpeg iFrame (that's adjustable bit rate) and uncompressed (8 and 10 bit). David AllinOneFilms.com

November 8th, 2009, 03:46 AM	#6
Marcello Mazzilli Major Player Join Date: Apr 2009 Location: ITALIA Posts: 416	I am trying Sheer but my need is not capturing because I capture directly with the camera. It could be usefull afterwards (archiving or switching from MAC to PC) but can't get it to work properly. I render my XDCAM MOV file in Sheer 10 bit. Works but.. 1) Can't get the presets in Premiere so (I think is for this) it goes slower than my original MOV 2) There is no batch feature. Would be usefull to convert all shot footage. Maybe I am missing something

November 8th, 2009, 01:00 PM	#9
Deke Kincaid Regular Crew Join Date: Jul 2007 Location: Los Angeles, CA Posts: 26	I think some of you guys are missing my original point. It was asking if the Flash XDR could do some type of lossless compression in hardware so we could use currently existing cards without having to resort to striping multiple cards together to get the speed. Uncompressed 422 for 1080p is beyond what even the Sandisk Extreme Pro does: 8 bit @ 1920 x 1080 @ 24fps = 95 MB per/sec 10 bit @ 1920 x 1080 @ 24fps = 127 MB per/sec 8 bit @ 1920 x 1080 @ 25fps = 99 MB per/sec 10 bit @ 1920 x 1080 @ 25fps = 132 MB per/sec 8 bit @ 1920 x 1080 @ 29.97fps = 119 MB per/sec 10 bit @ 1920 x 1080 @ 29.97fps = 158 MB per/sec

November 8th, 2009, 06:00 PM	#10
Mark Job Trustee Join Date: Sep 2006 Location: Montreal, Quebec, Canada Posts: 1,138	Hey Deke: I would like to see the Flash XDR have the choice to stripe 2 or 4 cards together. 2 x 90 MBps sandisk Extreme cards should give you enough sustained throughput to meet the specifications you listed. 4 x 90 MBps sandisk Extreme cards would certainly give you the sustained throughput to meet your stated criteria. I say Raid card number choice makes it a litlle more economical for me.

November 8th, 2009, 08:27 PM	#11
Deke Kincaid Regular Crew Join Date: Jul 2007 Location: Los Angeles, CA Posts: 26	Right.... but how do you then remove the cf cards from the flash xdr read the stripe on your windows/mac box so it will recognize it properly? I don't even know of any usb devices that let you do this. It just sounds very prone to screw ups when you in the field recording and either you mix up the pairs/quads or misplace a card. The only solutions I've seen are very clunky. One is a pci-express card that goes inside your machine that fits 2-4 cards. The other is a SSD box with a esata port on the back. Especially for in the field, the esata enclosure seems like option that will lead to the least amount of problems (wish it had 4 slots). The last thing I want to do is have to carry a full size pc in the field and then have to open up the inside of it just to swap out CF cards. Card Reader Internal Micro SATA UDD from Addonics CR2T - Compact Flash CF to 2.5? SATA Spanning / Mirroring Enclosure Do you know of any quad compact flash usb readers because that would be cool?