So I've downloaded this MSU Cartoonizer filter for VirtualDub.

Aside from never ever figuring out how to make Virtual Dub work, I've now made myself determined to do so because I want to do this cool cartoon effect.

How do I apply the filter and then export it with the filter settings?

I cannot figure this program out! And it pretty much freezes no matter what I ask it to do!

Have you looked at Silhouette Roto? http://www.silhouettefx.com/roto/

John, you may look into Imagineer Systems' Mocha, they have this incredible planar tracking/roto technology and you can download a trial version.