Capture uncompressed

Bill Anderson · August 17th, 2004, 10:27 AM

Thanks Joshua. That makes sense. I think I was being a bit overly optimistic. However, the FireStore drive is an excellent looking tool. And who knows, perhaps a FireStore "uncompressed" capture disc (and HD) will be in the future - that's optimism.
Cheers.

Aaron Shaw · August 17th, 2004, 10:57 AM

Most likely. Isn't the new JVS HDV ENG camera going to be able to record uncompressed to disk if one has the necessary accessories? I may be wrong as I can't remember where I heard that.

Graeme Nattress · August 17th, 2004, 10:58 AM

I don't think you're going to see Firestore capture uncompressed. First, a single drive isn't fast enough. Second, there's no advantage.

The "correct" place to convert DV from compressed to uncompressed is inside the NLE, at the time you need it converted. You can do this in FCP by dropping DV files onto an uncompressed timeline. This won't improve the DV - you need to upsample or reconstruct the chroma for that - see my website www.nattress.com for my experiments in that!

The way FCP works, you could just edit entirely in DV for speed, and then just change the final timeline to uncompressed and render. If you need to take any shots into AE or Shake or whatever, you can export them to an uncompressed codec at that point, although, if you don't need to keep an alpha channel, use PhotoJPEG @ 100% quality as it's imperceptably worse than Animation 100% (apart from the lack of alpha) and will work better over multiple generations than any "uncompressed' 4:2:2 based codec.

If you were to bump DV to uncompressed in an external hard drive recorder, the only thing you'd get would be to run out of hard disk space faster.

Graeme

Aaron Shaw · August 17th, 2004, 11:20 AM

I must disagree with Graeme. You gain a lot by recording the uncompressed signal. Most significantly you retain all of your color information and greatly improve the dynamic range of your signal. This is a very big bonus for those using DV for features and shorts.

Converting to uncompressed in a NLE timeline doesn't really do anything as it doesn't recreate the data which has been removed in the first place.

Graeme Nattress · August 17th, 2004, 11:28 AM

You've missed my point. On a DV camera you don't have access to an uncompressed signal, so there's none to record. All you can do is make the most of it in post.

AFAIK, there is NO uncompressed camcorder available - none. It's just not needed. Uncompressed is great in post when you're moving between apps and need to avoid generation loss, but during shooting, it's a very expensive luxury.

Graeme

Aaron Shaw · August 17th, 2004, 12:04 PM

Ah indeed I did. My apologies!

I know that there is some (near finished?) work on a mod of the DVX100A that will allow it to export uncompressed video straight off of the CCD array and several HD cameras in dev.

Joshua Starnes · August 17th, 2004, 12:50 PM

<<<-- Originally posted by Aaron Shaw : Ah indeed I did. My apologies!

I know that there is some (near finished?) work on a mod of the DVX100A that will allow it to export uncompressed video straight off of the CCD array and several HD cameras in dev. -->>>

And that's just about the only way to do it in DV. But considering the cost of the camera involved and the problems inherint in messing with the sensor, it's probably more worthwhile just to buy your own sensor, like Obin is doing.

Bill, if your worried about degradation in post for filmout, you should read (and for all I know, you already have) the American Cinematographer article on 28 Days Later.

As I remember it, they took all of their 'printed' takes, upsampled them to 4:2:2 and put them onto D1, which is what they worked on in post and for color correction and their final online edit. The final negative, I believe, was scanned out from the D1 tape as well - after capture they never went back to DV. That can be expensive, of course, but if film out is your goal, it might be something to look at.

Graeme Nattress · August 17th, 2004, 12:58 PM

For going out to film, you should boost to 4:4:4 via a reconstruction, rather than a linear interpolation technique - as I have proved, it looks superior. And 28days was over sharpened, and poorly uprezzed. You'd get better results up-rezzing to an HD format using modern techniques rather than going to D1, which is a 4:2:2 SD format. 28days is no best practise in this regard....

Graeme

Bill Anderson · August 17th, 2004, 02:27 PM

Interesting. I had no complaints with 28 DAYS LATER, considering it was shot on miniDV, but if there is a better way I don't have a problem with that! And I know Graeme is quite learned on this topic. Joshua, thanks, that is the way I understand this workflow too, but would like to add (you might find this obvious) that there's little point if any going back to DV after working in 4:2:2 or 4:4:4

Bill Anderson · August 17th, 2004, 02:32 PM

Graeme, where can I/we learn more about the 4:4:4 reconstruction method that you mentioned?

Graeme Nattress · August 17th, 2004, 03:25 PM

http://www.nattress.com/filmEffectsGNicerTests.htm

Shows the results from my filter for FCP. I'm working on an improved version now....

Graeme

Aaron Shaw · August 17th, 2004, 03:46 PM

Any chance your plugin will be ported to Premiere Pro? I would love to get my hands on a copy...

Steve Dent · August 19th, 2004, 04:31 PM

I think the original poster may have been talking about a method whereby using Avid software, and possibly others, you can capture 10-bit SDI video, uncompressed, through the firewire port of a computer.

However, you still have to have multiple drives, probably, to support that level of data rate.

Also, it's irrelevent with any DV camera I know, none of which support an SDI out signal. However, it would be relevent with DigiBeta cameras or others.

Some DV decks support SDI output as well. They transcode the DV signal from the tape to an upconverted SDI signal. This process, as far as I know, would not improve the quality of the original signal in any way, however, neither would it degrade it.

Technically, no upconverted DV signal would be "improved" since the software can't add information that wasn't there in the first place. However, it can be altered so that it looks better for large format display, using software discussed above.

Graeme Nattress · August 19th, 2004, 05:35 PM

I, personally, think that video sources of all qualities, from DV through DigiBeta to HD formats can be improved though the use of clever algorithms - indeed, that's my current mission in video programming!! I wouldn't be talking about the subject unless I passionately believed it to be so!

I think those who've seen the current results of my research can attest to the veracity of my claims, but I myself think that I can still improve on what I'm doing now as I get more used to working in this problem space.

When a deck converts DV to SDI, it just linear interpolates the chroma to produce the 4:2:2 signal from the 4:1:1 signal. You're right to say that this doesn't improve the quality at all - just represents it in a nicer fashion. I, however, have developed a method of intelligent luma based analysis that takes all the known information into account - luma and chroma, to make a much more accurate guess as to what the original 4:4:4 chroma was before it was compressed. This is not magicing information out of nowhere, but making the most possible use of all the information we have, by using the full resolution of the luma as a guide to help reconstruct the chroma. As far as I know, nobody else is doing it this way.

Similarly, I'm working on uprezzing DV to HD, and again, I'm not using standard interpolation filters like bicubic, or specialist ones like sinc or Lanczos, which, after all, don't produce the best of results even in the best of circumstances. They're slow, cause ringing and are fairly dumb in that they don't understand the nature of the individual picture they're uprezzing. Some have attempted to improve matters using content adaptive scaling, which, again, AFAIK, manipulates the interpolation parameters on a pixel by pixel basis to improve the results - and they do, to an extent. Algolith make a great scaler using these techniques, for instance. My technique is completely different and doesn't involve interpolation filters at all, and is, to my eyes, producing much better results. I've also seen some comparisons between it, and results from S-Spline which is often mentioned on these forums. I can safely say, that so far it's looking to be a lot better and faster than S-Spline, but again, that's my eyes!

So what I'm really saying, is that you can make DV look better than it is, but just capturing it uncompressed via SDI alone is not going to do this. To say the native DV image is as good as it gets is true enough under normal circumstances, but doesn't take into account the exciting developments in image processing that I and others are working on.

Graeme

Matt McEwen · August 20th, 2004, 09:48 AM

Hello,

I thought I would jump in on this thread...FireStore will only record an incoming DV25 signal from a FireWire 400 device such as a camcorder, switcher or deck. FireStore does not have a codec, nor does it have the ability to record and format other data streams other than DV25. The power of FireStore is that it is able to record a DV25 stream to a computer compatible FireWire drive in native NLE file formats. The processor and DTE Technology in FireStore makes it possible to write to disk in these native file formats. The end result is clips on disk that are identical to what your NLE would write during the capture process. This ensures that the clip is fully compatible with all the NLE's functions. Being on FireWire drives, it is possible to hot plug the HDD to Macs or PCs. All of this combines to eliminate the chore of ingesting your footage in the production process.

There are no plans currently for an uncompressed FireStore, mainly due to the fact that there is a small number of camcorders with uncompressed outputs and a lack of portable disk drives that are fast enough to record the stream. Remember, most FireStore products are portable devices designed to be used in the field.

But in the future, anything is possible. Focus has joined the HDV consortium and we hope to implement this format in FireStore products in the future. We are not yet sure in what form however. HDV is still in its early days.

On another note, the AJA Io box is the only device I am aware of that will transmit real-time SDI over FireWire 400. It of course, it requires a disk array to record the 270Mb/s stream and a Power Mac G4 or G5.

Hope that helps,
Matt McEwen

August 17th, 2004, 10:58 AM	#18
Graeme Nattress RED Problem Solver Join Date: Sep 2003 Location: Ottawa, Canada Posts: 1,365	I don't think you're going to see Firestore capture uncompressed. First, a single drive isn't fast enough. Second, there's no advantage. The "correct" place to convert DV from compressed to uncompressed is inside the NLE, at the time you need it converted. You can do this in FCP by dropping DV files onto an uncompressed timeline. This won't improve the DV - you need to upsample or reconstruct the chroma for that - see my website www.nattress.com for my experiments in that! The way FCP works, you could just edit entirely in DV for speed, and then just change the final timeline to uncompressed and render. If you need to take any shots into AE or Shake or whatever, you can export them to an uncompressed codec at that point, although, if you don't need to keep an alpha channel, use PhotoJPEG @ 100% quality as it's imperceptably worse than Animation 100% (apart from the lack of alpha) and will work better over multiple generations than any "uncompressed' 4:2:2 based codec. If you were to bump DV to uncompressed in an external hard drive recorder, the only thing you'd get would be to run out of hard disk space faster. Graeme __________________ www.nattress.com - filters for FCP

August 17th, 2004, 11:28 AM	#20
Graeme Nattress RED Problem Solver Join Date: Sep 2003 Location: Ottawa, Canada Posts: 1,365	You've missed my point. On a DV camera you don't have access to an uncompressed signal, so there's none to record. All you can do is make the most of it in post. AFAIK, there is NO uncompressed camcorder available - none. It's just not needed. Uncompressed is great in post when you're moving between apps and need to avoid generation loss, but during shooting, it's a very expensive luxury. Graeme __________________ www.nattress.com - filters for FCP

August 17th, 2004, 12:58 PM	#23
Graeme Nattress RED Problem Solver Join Date: Sep 2003 Location: Ottawa, Canada Posts: 1,365	For going out to film, you should boost to 4:4:4 via a reconstruction, rather than a linear interpolation technique - as I have proved, it looks superior. And 28days was over sharpened, and poorly uprezzed. You'd get better results up-rezzing to an HD format using modern techniques rather than going to D1, which is a 4:2:2 SD format. 28days is no best practise in this regard.... Graeme __________________ www.nattress.com - filters for FCP

August 17th, 2004, 03:25 PM	#26
Graeme Nattress RED Problem Solver Join Date: Sep 2003 Location: Ottawa, Canada Posts: 1,365	http://www.nattress.com/filmEffectsGNicerTests.htm Shows the results from my filter for FCP. I'm working on an improved version now.... Graeme __________________ www.nattress.com - filters for FCP

August 19th, 2004, 05:35 PM	#29
Graeme Nattress RED Problem Solver Join Date: Sep 2003 Location: Ottawa, Canada Posts: 1,365	I, personally, think that video sources of all qualities, from DV through DigiBeta to HD formats can be improved though the use of clever algorithms - indeed, that's my current mission in video programming!! I wouldn't be talking about the subject unless I passionately believed it to be so! I think those who've seen the current results of my research can attest to the veracity of my claims, but I myself think that I can still improve on what I'm doing now as I get more used to working in this problem space. When a deck converts DV to SDI, it just linear interpolates the chroma to produce the 4:2:2 signal from the 4:1:1 signal. You're right to say that this doesn't improve the quality at all - just represents it in a nicer fashion. I, however, have developed a method of intelligent luma based analysis that takes all the known information into account - luma and chroma, to make a much more accurate guess as to what the original 4:4:4 chroma was before it was compressed. This is not magicing information out of nowhere, but making the most possible use of all the information we have, by using the full resolution of the luma as a guide to help reconstruct the chroma. As far as I know, nobody else is doing it this way. Similarly, I'm working on uprezzing DV to HD, and again, I'm not using standard interpolation filters like bicubic, or specialist ones like sinc or Lanczos, which, after all, don't produce the best of results even in the best of circumstances. They're slow, cause ringing and are fairly dumb in that they don't understand the nature of the individual picture they're uprezzing. Some have attempted to improve matters using content adaptive scaling, which, again, AFAIK, manipulates the interpolation parameters on a pixel by pixel basis to improve the results - and they do, to an extent. Algolith make a great scaler using these techniques, for instance. My technique is completely different and doesn't involve interpolation filters at all, and is, to my eyes, producing much better results. I've also seen some comparisons between it, and results from S-Spline which is often mentioned on these forums. I can safely say, that so far it's looking to be a lot better and faster than S-Spline, but again, that's my eyes! So what I'm really saying, is that you can make DV look better than it is, but just capturing it uncompressed via SDI alone is not going to do this. To say the native DV image is as good as it gets is true enough under normal circumstances, but doesn't take into account the exciting developments in image processing that I and others are working on. Graeme __________________ www.nattress.com - filters for FCP

August 17th, 2004, 10:27 AM	#16
Bill Anderson Regular Crew Join Date: Jul 2004 Location: Ottawa. Canada Posts: 163	Thanks Joshua. That makes sense. I think I was being a bit overly optimistic. However, the FireStore drive is an excellent looking tool. And who knows, perhaps a FireStore "uncompressed" capture disc (and HD) will be in the future - that's optimism. Cheers.

August 17th, 2004, 10:57 AM	#17
Aaron Shaw Major Player Join Date: Jul 2004 Posts: 587	Most likely. Isn't the new JVS HDV ENG camera going to be able to record uncompressed to disk if one has the necessary accessories? I may be wrong as I can't remember where I heard that.

August 17th, 2004, 11:20 AM	#19
Aaron Shaw Major Player Join Date: Jul 2004 Posts: 587	I must disagree with Graeme. You gain a lot by recording the uncompressed signal. Most significantly you retain all of your color information and greatly improve the dynamic range of your signal. This is a very big bonus for those using DV for features and shorts. Converting to uncompressed in a NLE timeline doesn't really do anything as it doesn't recreate the data which has been removed in the first place.

August 17th, 2004, 12:04 PM	#21
Aaron Shaw Major Player Join Date: Jul 2004 Posts: 587	Ah indeed I did. My apologies! I know that there is some (near finished?) work on a mod of the DVX100A that will allow it to export uncompressed video straight off of the CCD array and several HD cameras in dev.

August 17th, 2004, 12:50 PM	#22
Joshua Starnes Major Player Join Date: Jun 2004 Location: Houston, TX Posts: 356	<<<-- Originally posted by Aaron Shaw : Ah indeed I did. My apologies! I know that there is some (near finished?) work on a mod of the DVX100A that will allow it to export uncompressed video straight off of the CCD array and several HD cameras in dev. -->>> And that's just about the only way to do it in DV. But considering the cost of the camera involved and the problems inherint in messing with the sensor, it's probably more worthwhile just to buy your own sensor, like Obin is doing. Bill, if your worried about degradation in post for filmout, you should read (and for all I know, you already have) the American Cinematographer article on 28 Days Later. As I remember it, they took all of their 'printed' takes, upsampled them to 4:2:2 and put them onto D1, which is what they worked on in post and for color correction and their final online edit. The final negative, I believe, was scanned out from the D1 tape as well - after capture they never went back to DV. That can be expensive, of course, but if film out is your goal, it might be something to look at.

August 17th, 2004, 02:27 PM	#24
Bill Anderson Regular Crew Join Date: Jul 2004 Location: Ottawa. Canada Posts: 163	Interesting. I had no complaints with 28 DAYS LATER, considering it was shot on miniDV, but if there is a better way I don't have a problem with that! And I know Graeme is quite learned on this topic. Joshua, thanks, that is the way I understand this workflow too, but would like to add (you might find this obvious) that there's little point if any going back to DV after working in 4:2:2 or 4:4:4

August 17th, 2004, 02:32 PM	#25
Bill Anderson Regular Crew Join Date: Jul 2004 Location: Ottawa. Canada Posts: 163	Graeme, where can I/we learn more about the 4:4:4 reconstruction method that you mentioned?

August 17th, 2004, 03:46 PM	#27
Aaron Shaw Major Player Join Date: Jul 2004 Posts: 587	Any chance your plugin will be ported to Premiere Pro? I would love to get my hands on a copy...

August 19th, 2004, 04:31 PM	#28
Steve Dent New Boot Join Date: Jun 2004 Posts: 8	I think the original poster may have been talking about a method whereby using Avid software, and possibly others, you can capture 10-bit SDI video, uncompressed, through the firewire port of a computer. However, you still have to have multiple drives, probably, to support that level of data rate. Also, it's irrelevent with any DV camera I know, none of which support an SDI out signal. However, it would be relevent with DigiBeta cameras or others. Some DV decks support SDI output as well. They transcode the DV signal from the tape to an upconverted SDI signal. This process, as far as I know, would not improve the quality of the original signal in any way, however, neither would it degrade it. Technically, no upconverted DV signal would be "improved" since the software can't add information that wasn't there in the first place. However, it can be altered so that it looks better for large format display, using software discussed above.

August 20th, 2004, 09:48 AM	#30
Matt McEwen Focus Enhancements Join Date: Oct 2003 Location: Campbell, CA. Posts: 508	Hello, I thought I would jump in on this thread...FireStore will only record an incoming DV25 signal from a FireWire 400 device such as a camcorder, switcher or deck. FireStore does not have a codec, nor does it have the ability to record and format other data streams other than DV25. The power of FireStore is that it is able to record a DV25 stream to a computer compatible FireWire drive in native NLE file formats. The processor and DTE Technology in FireStore makes it possible to write to disk in these native file formats. The end result is clips on disk that are identical to what your NLE would write during the capture process. This ensures that the clip is fully compatible with all the NLE's functions. Being on FireWire drives, it is possible to hot plug the HDD to Macs or PCs. All of this combines to eliminate the chore of ingesting your footage in the production process. There are no plans currently for an uncompressed FireStore, mainly due to the fact that there is a small number of camcorders with uncompressed outputs and a lack of portable disk drives that are fast enough to record the stream. Remember, most FireStore products are portable devices designed to be used in the field. But in the future, anything is possible. Focus has joined the HDV consortium and we hope to implement this format in FireStore products in the future. We are not yet sure in what form however. HDV is still in its early days. On another note, the AJA Io box is the only device I am aware of that will transmit real-time SDI over FireWire 400. It of course, it requires a disk array to record the 270Mb/s stream and a Power Mac G4 or G5. Hope that helps, Matt McEwen