upsampling / supersampling
 

I have been thinking on this for a while and wanted to see if you guys had any ideas.

After watching some footage on the Discovery channel of high-detail landscapes and backgrounds, I thought about how that is basically 720x480, which is what my camera captures at, but the detail is a lot better looking than a standard DV camera. Of course, this has a lot to do with the optics and how many pixels are being sampled up front. (Or if it is shot on film and transferred to video, it's automatically better.)

So I looked into the theory of upsampling images. Everyone knows that you can shrink a digital image with no problem, but when you blow one up, even using the best algorithms such as Lancoz(sp?), b-spline, and bicubic sampling, you'll still introduce a lot of softness to the image.

One approach for you Photoshop guys, is to take your 720x480 frame and upsize it in small increments, several times. This is better than doing it all in one step.

Now you do all your hue/saturation, curves adjustments and run the unsharp mask filter (which is easy to overdo, if you are running it on tye 720x480 image. ), since you would have more image data to manipulate.

Now when you take the final image and size back down to a 720x480 frame, you should have a better looking image.

Of course, this may be prohibitive since you would have to export your footage as a sequence of frames and run this one frame at a time using a Photoshop action script, but it may be worth it if you only use it on, say, landscapes.

This I have yet to try, but one thing I have done was to take a scanned 35mm picture, add grain, and render to a 720x480 avi, and it looked like footage. (Of course, you can't do this on waterfall scenes.)

Anyhow, I'm just throwing some ideas out there. Let me know what you think.

,Frank

>So I looked into the theory of upsampling images. Everyone >knows that you can shrink a digital image with no problem, but >when you blow one up, even using the best algorithms such as >Lancoz(sp?), b-spline, and bicubic sampling,

Actually, image shrinking suffers similar problems from aliasing which adds noise to the picture.

None of these upsampling algorithms listed above are particularly good. They are linear combinations of a small number of pixels and thus are popular because they are computationally simple for the job they do. Since few samples are required, the phase shift is also small. Larger filters will shift the image by 2 or more pixels, requiring extra processing to restore the result. Ideally, you would like to use a 2 dimensional sinc function sin(x)/x interpolation during the resampling, although this computationally intensive algorithm will take more time. However, at least 8 or so points are needed before the polynomial interpolators are bested.
With even more points, the polynomial interpolators fall behind
quickly.

>you'll still introduce >a lot of softness to the image.

If someone showed you the frequency response of these interpolators, you would gag. There will always be some softness as you zoom in because you are trying to view absent information in a higher resolution format. However, these algorithms readuce the original information.

Since I work in audio, I'll use an audio example. Say you have a telephone recording, which has a sampling rate of 8000 Hz,
a theoretical bandwidth of 4000 Hz. (However, the real bandwidth is somewhat smaller). Even if you record your telephone line on a 48000 Hz digital recorder and play it back on a fantastic stereo system, the recording will have no more high frequency content than that in the original recording. If you want to make it brighter, you can use an equalizer to boost frequencies
below 4000 Hz. The equalizer won't work above 4000 Hz because there are no frequencies present. However, if one of the bicubic methods were used to convert the 8000 Hz sampling rate telephone signal to a sampling rate of 48000 Hz, frequencies in the 2000-4000 range would be tapered off, resulting in a recording that is singificantly duller than the original. Although audio and video signals enter our bodies using different sensors,
the signals are processed in a similar manner.

>One approach for you Photoshop guys, is to take your 720x480 >frame and upsize it in small increments, several times. This is >better than doing it all in one step.

This is disturbing. If Photoshop users are doing multi-pass resampling, then Adobe should offer higher order interpolators so that only one step is needed. They are wasting your time.

Can it be that Discovery channel cameras do not have optical anti-aliasing? Then what you see is many aliasing artifacts from the details that can not be completely resolved. Visually it could produce detail-reach picture, even though there is no actual resolution there.
On DV camera optical anti-aliasing basically removes all the details that can not be resolved by sensor. Mathematically such an approach is more correct, but it's percepted as a lack of details. Discovery might employ mathematically incorrect, but visulaly more pleasing effects.

I can't imagine how upsampling an image and then down-sampling
it will INCREASE the quality. It may alter the look of the image to
something that you find more interesting.

I've seen some good results upsampling images using fractal
algorithms. Although these are very very time consuming
algorithms.

Rob: I was thinking that not just upsampling, then downsampling, but upsampling, adjusting, then downsampling would be better, since curves adjustments, etc would have more pixels to work with when the image is larger. True, it probably wouldn't increase the quality, but maybe make it appear to look better.

Vladimir: I haven't heard a lot about optical anti-aliasing. I will look into this more.

Gints: Very thorough answer there. Thanks for explaining that!

I've seen the technique Frank is talking about. It works very effectively in still photography. But it really is just tricking the mind and eye into perceiving more detail than what is actually in the image. I don't think it would be a viable tool for video production because of the length rendering times.

Jeff: Yes, it would be very cumbersome to run this process on each frame. It would maybe be something you would do with just a particular piece of footage, like a landscape. Even that would be time consuming, though.

I was wondering - when DV footage is transferred to film, what sort of process do they run on the footage to make it look better when it is blown up?

I would suspect they use unsharp mask to make the images appear sharper to the eye. There are some new sharpening filters (software) coming soon that should advance the quality and speed of operation.

The Discovery Channel also does a great deal of computer animation and the realism is amazing on some of the scenes.

There are also fractal-based image 'magnification' techniques used in digital photography. I don't recall any specific products, but a few years ago when CCD pixels were very expensive, these programs seemed to be popular.

I have heard complaints about how DVDs are being over sharpened, showing the white/black 'ringing' effect on the bigger TVs.

Most viewers set the sharpness control on their TV's to the highest setting. This creates many visual problems for DVD's and other low resolution mediums. It should really be set to the lowest setting.

>I've seen the technique Frank is talking about. It works very >effectively in still photography. But it really is just tricking the >mind and eye into perceiving more detail than what is actually in >the image. I don't think it would be a viable tool for video >production because of the length rendering times.

Although I'm new at videography, I used to experiment with image processing. I noticed that some aliasing during resizing, expecially in black&white images, appeared preferable to none at all. The aliasing is most pronounced around edges (which are high frequencies), and so this noise results in an increase in image sharpness or more information around the edges.

For video, I suspect this aliasing noise around the edges would animate (like that found in line "jaggies") and thus be as undesirable.

>Rob: I was thinking that not just upsampling, then >downsampling, but upsampling, adjusting, then downsampling >would be better, since curves adjustments, etc would have >more pixels to work with when the image is larger. True, it >probably wouldn't increase the quality, but maybe make it >appear to look better.

Frank,

Provided the filters used to upsample and downsample are pretty good, more image fidelity would be preserved if upsampled, processed, then downsampled. Operating at higher sampling rates allow more bandwidth for out-of-band or aliased components. Non-linear effects such as color curves often introduce out-of-band components. Most video processors accept these artifacts as a fact of life. However, much of that could be reduced by processing at a higher samping rate to reduce the aliased energy by spreading it into the upper frequencies. These undesirable components which would then be filtered out when the signal is downsampled. Without the extra bandwidth, the aliasing components spill into the baseband and ccould never be eliminated. Generally, these aliasing components are part of random, animated grunge one sees at edges.

Audio signal processing has found significant quality improvements by operating at higher sampling frequencies.
Basic guitar distortion called waveshaping is quite similar to video's color curves. In Waveshaping, a non-linear curve is applied to the audio signal to remap it. The effect is harmonic generation and the results are usually easy to hear.

Gints: That is very interesting!

In theory, if an image was converted to, say, 48 bits, manipulated, and then dropped to 24, would that work better than manipulating in 24 bit mode, or would the footage have had to originate in 48 bit mode?

I know a lot of the newer applications are taking advantage of 48 bit editing.