Frame mode/16 X 9 Resolution Loss

Josh Bass · September 22nd, 2003, 01:57 AM

Okay. . .seems I understand what you guys are saying now. I will try the tests, just the same. I looked at it on my NTSC monitor (not a good enough test?) and they looked the same to me.

Now, let us all hope and pray for our 24p, HD, true 16:9 XL2 to come out. . .come on, bow your heads. . .okay, you're not all bowing. . .ah. . .there. Much better.

Hugh DiMauro · September 22nd, 2003, 07:33 AM

I thank you all for the passionate responses. Over the weekend, after my original post, I took my XL1s to a scenic location and shot 15 minutes of video in interlaced mode with the 16 x 9 guides. Upon arriving home, I played the raw footage on a JVC high resolution monitor. Then, via fire wire, I loaded the footage onto my computer utilizing Vegas 4.0 with the properties set to deinterlace. Furthermore, I activated the pan/crop tool, and chose the 16 x 9 letterbox mask from the drop down menu. Here are my findings:

1) The raw footage viewed on my hi resolution JVC monitor looked fabulous even though it was interlaced. It had a soft, filmic quality even though it lacked that "progressive scan" look. I was almost convinced to just shoot all my short features interlaced.

2) For some reason, even though my Vegas 4.0 properties were set to de-interlace, the footage still looked interlaced and when I tried a frame capture, the image shook as if I had made an interlaced frame capture. Rats.

3) The letterboxing mask on the pan/crop dropdown menu did a FABULOUS (and I mean FABULOUS) job.

So, gentle friends, based partially on my own tests and the priceless advice offered by my DV INFO colleagues who had rigorously tested these theories prior to my quest for answers, my wedding videos will be shot entirely in FRAME mode and my indepedent films will be shot entirely in FRAME mode with 16 x 9 guides to facilitate masking in post.

NOW... here's stupid question #436: Did I read on this thread that applying a letterbox mask in post (specifically with the pan/crop tool on Vegas) reduces picture quality/resolution? I kinda thought that the mask just innocently covered the top and bototm of the screen.

HAAAALLLLLP!

Josh Bass · September 22nd, 2003, 11:55 AM

I think what they mean by that is that you're throwing away the information at the top and bottom of the 4:3 frame when you apply the letterbox in post, thereby reducing the total number of lines of resolution being used in your picture by 25%. . .not that the portion you WANT to see becomes less sharp.

The in-camera anamorphic mode squeezes the pixes from the top and bottom into that 16:9 portion, so it makes use of them instead of chopping them away.

Hugh DiMauro · September 22nd, 2003, 01:38 PM

Thank you. That clarified it. I mean, who cares if you cover up pixels as long as the portion that shows looks crisp and sharp, right? I will continue to shoot frame mode with the viewfinder guides then mask in post.

Once again, excellent job on your clay animation short, boychic!

L'Chaim

Josh Bass · September 22nd, 2003, 04:27 PM

Thanks. That's "award winning" claymation short. About anamorphic, seems more pixels = more resolution, so I'll do the tests when I stop being lazy, and give a final ruling.

Boyd Ostroff · September 22nd, 2003, 09:17 PM

<<<-- Originally posted by Josh Bass :The in-camera anamorphic mode squeezes the pixes from the top and bottom into that 16:9 portion, so it makes use of them instead of chopping them away. -->>>

Sorry John, you lost me there. You would need a higher resolution CCD to do that since the field of view would need to get wider in order to use those pixels at the top and bottom. Otherwise you wouldn't end up with the correct 16:9 proportion. The only way to use those pixels on the XL1s would be with an anamorphic lens.

In-camera 16:9 works by first cropping the image to 720x360, discarding pixels from the top and bottom. Then it stretches the result back to 720x480 to create an anamorphic image. So the image that's recorded is indeed 480 lines high, but the vertical resolution will only be 360 lines.

Josh Bass · September 22nd, 2003, 11:23 PM

So then it DOES do the same thing you're doing when cropping in post. . .albeit with a lower resolution. . .how could this possibly be so confusing?

Hugh DiMauro · September 23rd, 2003, 07:35 AM

No no! I understand! I understand what Boyd said. It's amazing that Canon engineers it's 16 x 9 that way. What comes to mind is that old joke: How many idiots does it take to screw in a lightbulb? TEN. One to hold the bulb and nine to lift the house and turn it. Why crop, then fill the screen again? It almost seems like Canon does that on purpose to "zetz" us! makes no sense.

Victor Muh · September 25th, 2003, 08:04 PM

Isn't another drawback in letterboxing in post the corruption of the DV data?

From what I understand, DV only maintains its quality if you don't touch it during editing. As soon as you add titling, adjust color or anything else, you get a drop in quality.

Barry Green · September 26th, 2003, 12:21 AM

<<<-- Originally posted by Hugh DiMauro : Why crop, then fill the screen again? -->>>

Because you have to change the shape of the video, or it isn't 16:9.

Look, all DV is 720 x 480. That's mandatory, that's what's regulated by the DV spec. It's all 720 x 480 (speaking strictly NTSC here). 4:3 video is 720 x 480. 16:9 video is 720 x 480. What changes to make it be widescreen? The SHAPE of the pixels. The # of pixels doesn't change, but the shape does.

In 4:3, the pixels are slightly tall and skinny. In 16:9, the pixels are slightly short and wide.

So to execute in-camera 16:9, the camera will extract a 16:9-shaped patch from the center of the CCD (which is a 720 x 360 piece of video) and then stretch it to fill the full 720 x 480 frame (because it must be 720 x 480, the laws of DV demand it). The result is video that would look quite squished horizontally if you tried to view it on a 4:3 TV, it requires a widescreen 16:9 TV to be displayed properly.

Barry Green · September 26th, 2003, 12:24 AM

<<<-- Originally posted by Victor Muh : Isn't another drawback in letterboxing in post the corruption of the DV data?

From what I understand, DV only maintains its quality if you don't touch it during editing. As soon as you add titling, adjust color or anything else, you get a drop in quality. -->>>

Yes, that is technically correct. But the question is, does the in-camera 16:9 damage the video quality more than the post-production uncompress/16:9 stretch/recompress process? In Canon cameras, they do a really pretty good job of the 16:9 process, so even though there is necessarily a resolution loss, it's not too bad, and the stretch is performed before compression. If you know you will want 16:9, and you're shooting on a Canon XL1/GL2, it's probably best to just go ahead and use the in-camera mode.

On a Sony VX2000, it's a different story. The in-camera 16:9 mode is vastly inferior, so you will actually get much better results if you perform the 16:9 conversion in post (assuming you use an excellent resize algorithm, such as provided by Vegas 4.0). Even though there's a decompress and recompress cycle, the results are still much cleaner than the in-camera 16:9 process on that camera.

Hugh DiMauro · September 26th, 2003, 06:12 AM

Oh man! That sucks! I just started shotting my movie in 4 x 3 with the electronic guides and I cropped in post. I'd HATE to have to re-shoot everything in electronic widescreen. Rats!

Hugh DiMauro · September 26th, 2003, 06:14 AM

Oh, by the way, when I letterbox in post, I am just applying a mask in Vegas 4.0 through the pan/crop tool. Are you saying that when I do it that way, it is stretching and squeezing and fondling and all of that? Or is it just covering the top and bottom of the frame? Remember, I'm doing it with one mouse stroke: pan/crop 16 x 9 dropdown letterboxing.

Josh Bass · September 26th, 2003, 12:03 PM

I've heard that Sonic Foundry's codec is really . . .um. . .good, and you lose a lot less with it when tweaking (letterbox, color correction, etc) than with some other codecs. Personally, I don't notice any quality loss from letterboxing after the fact. Granted, I'm looking on my NTSC monitor (14 inches) or a small TV. . .never seen it on a real big screen. I believe as long you render only once, that is, apply all your effects, tweaks, blips, bloops and blops to the raw footage and then render, as opposed to applying some tweaks, then rendering, then applying bloops and blips and re-rendering, you'll be okay. Only time I ever saw a serious loss in quality is when I converted 60i to 24p.

Barry Green · September 26th, 2003, 05:18 PM

Hugh, you're probably fine, don't even worry about it, you won't have to re-shoot anything. If you're looking for actual anamorphic footage, and you're doing it in post, Vegas is a great place to do it... I'm sure the results of stretching the footage will be competitive with in-camera 16:9.

As far as what you're doing in Vegas, with the drop-down 16:9, if your project has an overall aspect ratio of 4:3, then all that does is add the black bars. If you change your project to be 16:9, then Vegas will automatically stretch the footage to be 16:9.

September 22nd, 2003, 01:57 AM	#16
Josh Bass Slash Rules! Join Date: Apr 2002 Location: Houston, Texas Posts: 5,472	Okay. . .seems I understand what you guys are saying now. I will try the tests, just the same. I looked at it on my NTSC monitor (not a good enough test?) and they looked the same to me. Now, let us all hope and pray for our 24p, HD, true 16:9 XL2 to come out. . .come on, bow your heads. . .okay, you're not all bowing. . .ah. . .there. Much better. __________________ The Bass' camerawork The Bass on IMDB

September 22nd, 2003, 07:33 AM	#17
Hugh DiMauro Trustee Join Date: Jun 2003 Location: Central Florida Posts: 1,205	Thank you all! I thank you all for the passionate responses. Over the weekend, after my original post, I took my XL1s to a scenic location and shot 15 minutes of video in interlaced mode with the 16 x 9 guides. Upon arriving home, I played the raw footage on a JVC high resolution monitor. Then, via fire wire, I loaded the footage onto my computer utilizing Vegas 4.0 with the properties set to deinterlace. Furthermore, I activated the pan/crop tool, and chose the 16 x 9 letterbox mask from the drop down menu. Here are my findings: 1) The raw footage viewed on my hi resolution JVC monitor looked fabulous even though it was interlaced. It had a soft, filmic quality even though it lacked that "progressive scan" look. I was almost convinced to just shoot all my short features interlaced. 2) For some reason, even though my Vegas 4.0 properties were set to de-interlace, the footage still looked interlaced and when I tried a frame capture, the image shook as if I had made an interlaced frame capture. Rats. 3) The letterboxing mask on the pan/crop dropdown menu did a FABULOUS (and I mean FABULOUS) job. So, gentle friends, based partially on my own tests and the priceless advice offered by my DV INFO colleagues who had rigorously tested these theories prior to my quest for answers, my wedding videos will be shot entirely in FRAME mode and my indepedent films will be shot entirely in FRAME mode with 16 x 9 guides to facilitate masking in post. NOW... here's stupid question #436: Did I read on this thread that applying a letterbox mask in post (specifically with the pan/crop tool on Vegas) reduces picture quality/resolution? I kinda thought that the mask just innocently covered the top and bototm of the screen. HAAAALLLLLP!

September 22nd, 2003, 11:55 AM	#18
Josh Bass Slash Rules! Join Date: Apr 2002 Location: Houston, Texas Posts: 5,472	I think what they mean by that is that you're throwing away the information at the top and bottom of the 4:3 frame when you apply the letterbox in post, thereby reducing the total number of lines of resolution being used in your picture by 25%. . .not that the portion you WANT to see becomes less sharp. The in-camera anamorphic mode squeezes the pixes from the top and bottom into that 16:9 portion, so it makes use of them instead of chopping them away. __________________ The Bass' camerawork The Bass on IMDB

September 22nd, 2003, 01:38 PM	#19
Hugh DiMauro Trustee Join Date: Jun 2003 Location: Central Florida Posts: 1,205	Thanks Josh Thank you. That clarified it. I mean, who cares if you cover up pixels as long as the portion that shows looks crisp and sharp, right? I will continue to shoot frame mode with the viewfinder guides then mask in post. Once again, excellent job on your clay animation short, boychic! L'Chaim

September 22nd, 2003, 04:27 PM	#20
Josh Bass Slash Rules! Join Date: Apr 2002 Location: Houston, Texas Posts: 5,472	Thanks. That's "award winning" claymation short. About anamorphic, seems more pixels = more resolution, so I'll do the tests when I stop being lazy, and give a final ruling. __________________ The Bass' camerawork The Bass on IMDB

September 22nd, 2003, 09:17 PM	#21
Boyd Ostroff Wrangler Join Date: Dec 2002 Location: Mays Landing, NJ Posts: 11,787	<<<-- Originally posted by Josh Bass :The in-camera anamorphic mode squeezes the pixes from the top and bottom into that 16:9 portion, so it makes use of them instead of chopping them away. -->>> Sorry John, you lost me there. You would need a higher resolution CCD to do that since the field of view would need to get wider in order to use those pixels at the top and bottom. Otherwise you wouldn't end up with the correct 16:9 proportion. The only way to use those pixels on the XL1s would be with an anamorphic lens. In-camera 16:9 works by first cropping the image to 720x360, discarding pixels from the top and bottom. Then it stretches the result back to 720x480 to create an anamorphic image. So the image that's recorded is indeed 480 lines high, but the vertical resolution will only be 360 lines.

September 22nd, 2003, 11:23 PM	#22
Josh Bass Slash Rules! Join Date: Apr 2002 Location: Houston, Texas Posts: 5,472	So then it DOES do the same thing you're doing when cropping in post. . .albeit with a lower resolution. . .how could this possibly be so confusing? __________________ The Bass' camerawork The Bass on IMDB

September 23rd, 2003, 07:35 AM	#23
Hugh DiMauro Trustee Join Date: Jun 2003 Location: Central Florida Posts: 1,205	No no! I understand! I understand what Boyd said. It's amazing that Canon engineers it's 16 x 9 that way. What comes to mind is that old joke: How many idiots does it take to screw in a lightbulb? TEN. One to hold the bulb and nine to lift the house and turn it. Why crop, then fill the screen again? It almost seems like Canon does that on purpose to "zetz" us! makes no sense.

September 25th, 2003, 08:04 PM	#24
Victor Muh Regular Crew Join Date: Aug 2002 Location: Paris, France Posts: 126	Isn't another drawback in letterboxing in post the corruption of the DV data? From what I understand, DV only maintains its quality if you don't touch it during editing. As soon as you add titling, adjust color or anything else, you get a drop in quality. __________________ Strive not to be a success, but rather to be of value. - Albert Einstein http://advaloreminternational.com

September 26th, 2003, 12:21 AM	#25
Barry Green Barry Wan Kenobi Join Date: Jul 2003 Location: North Carolina Posts: 3,863	<<<-- Originally posted by Hugh DiMauro : Why crop, then fill the screen again? -->>> Because you have to change the shape of the video, or it isn't 16:9. Look, all DV is 720 x 480. That's mandatory, that's what's regulated by the DV spec. It's all 720 x 480 (speaking strictly NTSC here). 4:3 video is 720 x 480. 16:9 video is 720 x 480. What changes to make it be widescreen? The SHAPE of the pixels. The # of pixels doesn't change, but the shape does. In 4:3, the pixels are slightly tall and skinny. In 16:9, the pixels are slightly short and wide. So to execute in-camera 16:9, the camera will extract a 16:9-shaped patch from the center of the CCD (which is a 720 x 360 piece of video) and then stretch it to fill the full 720 x 480 frame (because it must be 720 x 480, the laws of DV demand it). The result is video that would look quite squished horizontally if you tried to view it on a 4:3 TV, it requires a widescreen 16:9 TV to be displayed properly.

September 26th, 2003, 12:24 AM	#26
Barry Green Barry Wan Kenobi Join Date: Jul 2003 Location: North Carolina Posts: 3,863	<<<-- Originally posted by Victor Muh : Isn't another drawback in letterboxing in post the corruption of the DV data? From what I understand, DV only maintains its quality if you don't touch it during editing. As soon as you add titling, adjust color or anything else, you get a drop in quality. -->>> Yes, that is technically correct. But the question is, does the in-camera 16:9 damage the video quality more than the post-production uncompress/16:9 stretch/recompress process? In Canon cameras, they do a really pretty good job of the 16:9 process, so even though there is necessarily a resolution loss, it's not too bad, and the stretch is performed before compression. If you know you will want 16:9, and you're shooting on a Canon XL1/GL2, it's probably best to just go ahead and use the in-camera mode. On a Sony VX2000, it's a different story. The in-camera 16:9 mode is vastly inferior, so you will actually get much better results if you perform the 16:9 conversion in post (assuming you use an excellent resize algorithm, such as provided by Vegas 4.0). Even though there's a decompress and recompress cycle, the results are still much cleaner than the in-camera 16:9 process on that camera.

September 26th, 2003, 06:12 AM	#27
Hugh DiMauro Trustee Join Date: Jun 2003 Location: Central Florida Posts: 1,205	Oh man! That sucks! I just started shotting my movie in 4 x 3 with the electronic guides and I cropped in post. I'd HATE to have to re-shoot everything in electronic widescreen. Rats!

September 26th, 2003, 06:14 AM	#28
Hugh DiMauro Trustee Join Date: Jun 2003 Location: Central Florida Posts: 1,205	Oh, by the way, when I letterbox in post, I am just applying a mask in Vegas 4.0 through the pan/crop tool. Are you saying that when I do it that way, it is stretching and squeezing and fondling and all of that? Or is it just covering the top and bottom of the frame? Remember, I'm doing it with one mouse stroke: pan/crop 16 x 9 dropdown letterboxing.

September 26th, 2003, 12:03 PM	#29
Josh Bass Slash Rules! Join Date: Apr 2002 Location: Houston, Texas Posts: 5,472	I've heard that Sonic Foundry's codec is really . . .um. . .good, and you lose a lot less with it when tweaking (letterbox, color correction, etc) than with some other codecs. Personally, I don't notice any quality loss from letterboxing after the fact. Granted, I'm looking on my NTSC monitor (14 inches) or a small TV. . .never seen it on a real big screen. I believe as long you render only once, that is, apply all your effects, tweaks, blips, bloops and blops to the raw footage and then render, as opposed to applying some tweaks, then rendering, then applying bloops and blips and re-rendering, you'll be okay. Only time I ever saw a serious loss in quality is when I converted 60i to 24p. __________________ The Bass' camerawork The Bass on IMDB

September 26th, 2003, 05:18 PM	#30
Barry Green Barry Wan Kenobi Join Date: Jul 2003 Location: North Carolina Posts: 3,863	Hugh, you're probably fine, don't even worry about it, you won't have to re-shoot anything. If you're looking for actual anamorphic footage, and you're doing it in post, Vegas is a great place to do it... I'm sure the results of stretching the footage will be competitive with in-camera 16:9. As far as what you're doing in Vegas, with the drop-down 16:9, if your project has an overall aspect ratio of 4:3, then all that does is add the black bars. If you change your project to be 16:9, then Vegas will automatically stretch the footage to be 16:9.