Rotoscoping software idea -- does this exist? at DVinfo.net

Go Back   DV Info Net > The Tools of DV and HD Production > Open DV Discussion

Open DV Discussion
For topics which don't fit into any of the other categories.


Reply
 
Thread Tools Search this Thread
Old January 5th, 2007, 12:40 PM   #1
Regular Crew
 
Join Date: Dec 2003
Location: DC, USA
Posts: 97
Rotoscoping software idea -- does this exist?

I had an idea which would make certain rotoscoping so much simpler, and I was wondering if this idea has already been implemented in a software package one can purchase.

The idea is that the camera sits on a tripod, motionless, and films the background without the subject. The subject is then filmed in front of the background doing whatever is necessary for the take. The software would then compare each frame which contains the subject with the footage that does not contain the subject, and through a series of tests such as examining the color differences between the frames, would be able to isolate the subject from the background. This would allow one to use green screen effects without a green screen, do digital DOF effects (applying a gaussian blur to the background but leave the subject unchanged), etc.

I have no idea how well it would work in practice, but it doesn't seem like it would be difficult to implement. Has this been done, and if not, is there any reason why it wouldn't work?
__________________
http://duffx.com
James Duffy is offline   Reply With Quote
Old January 5th, 2007, 01:08 PM   #2
Major Player
 
Join Date: Feb 2004
Location: Philadelphia, PA, USA
Posts: 548
Yes it exists ... it's a difference key operation in most compositing apps (and some NLE programs) ... and it typically doesn't work very well.

Reasons why it doesn't work include noise (every pixel is a little different, all the time) shadows (it's still the background, but it's different than before) and color similarities between foreground and background (if you have a scene that calls for everything to be lit with an eery blue light ... forground and background will have the same color pallet)
__________________
Nick Jushchyshyn Matchmoving, Compositing, TD
imdb
Nick Jushchyshyn is offline   Reply With Quote
Old January 5th, 2007, 01:38 PM   #3
Regular Crew
 
Join Date: Dec 2003
Location: DC, USA
Posts: 97
I was thinking something more advanced than a single operation, like subtracting one image from another. That's why I was thinking you film the background without the subject and not just have a screen grab of it, so the software could account for the camera's natural grain. Color difference alone wouldn't account for it, since skin tone could match shades of the background, but I was thinking a more advanced algorithm that used a combination of tests for edge detection to identify moving objects, and show only the color information from these moving objects.

Again, not sure if it would work properly, just an idea. If you close one eye, you can still identify a clear boundary where one object ends and another begins. Of course, maybe to work properly it would require significant advances in artificial intelligence and computer vision.
__________________
http://duffx.com
James Duffy is offline   Reply With Quote
Old January 5th, 2007, 01:52 PM   #4
Major Player
 
Join Date: Feb 2004
Location: Philadelphia, PA, USA
Posts: 548
Mmmm ... that sounds like optical flow...
http://www.fxguide.com/article333.html
__________________
Nick Jushchyshyn Matchmoving, Compositing, TD
imdb
Nick Jushchyshyn is offline   Reply With Quote
Old January 5th, 2007, 02:29 PM   #5
Regular Crew
 
Join Date: Dec 2003
Location: DC, USA
Posts: 97
Hmm, optical flow looks to me to be a combination of Realsmart Twixtor, 2d3 Steadymove, and WinMorph, although probably more powerful/flexible. I'm not familiar enough with the technology behind it to know if it could be adapted to extract moving objects from a still background.

I did find this: http://www.dvxuser.com/V6/showthread.php?t=59238
He seems to be detailing the same thing I'm describing, but using an advanced technology in-camera to assign a z-index (depth) to each pixel, which could then be used in post to apply effects to objects within depth-ranges. That would be amazing, and much cleaner than what I'm suggesting, but it would also require new technology to be doable. I'm wondering if it's possible to do with current video technology through motion detection of 2-d images.

Well either way, I can't wait for the day when the technology behind his idea is implemented.
__________________
http://duffx.com
James Duffy is offline   Reply With Quote
Old January 5th, 2007, 04:15 PM   #6
Trustee
 
Join Date: Mar 2004
Location: Milwaukee, WI
Posts: 1,719
The grain however wouldn't match up. If noise was constant it may have a fighting chance but noise changes all the time. That means it would be almost impossible to match up the same noise pattern between the foreground plate and the background plate.

The other reason why this doesn't work very well is because people are not soilid color objects and chances are that a few or more pixels will match the color of the background plate. This would mean you would end up with holes in your subject. This is why blue and green screens are used because it is easy to setup the subject so they do not share any of the same chroma information as the screen.

A keyer of this type would have to have a butt load of error correction built in to try and compensate for the subject being over similar colors.

For example I mean lets say your subjects is wearing a brown coat and they walk in front of a brown fence. Maybe not at all of brown pixels will give you trouble but some of them will.

It can be done in a pinch but you would end up rotoscoping the errors so much that you might end up just rotoscoping from the start.

This method has been used in the past with keyers such as Ultimate. There was a screen correction filter that was used to clean up badly lit bluescreens. What you would do shoot a clean plate of the bluescreen and then shoot your subject in front of the bluescreen. You then used the filter to clean up the quality of the bluescreen. This was not a keyer on it's own but just a way to help get a cleaner key.
Thomas Smet is offline   Reply With Quote
Old January 6th, 2007, 03:45 AM   #7
Inner Circle
 
Join Date: Feb 2004
Location: switzerland
Posts: 2,131
you described exactly what MOKEY does.
Welcome on earth...
Giroud Francois is offline   Reply With Quote
Old January 6th, 2007, 10:44 AM   #8
Trustee
 
Join Date: Feb 2004
Location: Suwanee, GA
Posts: 1,241
Quote:
Originally Posted by Giroud Francois
you described exactly what MOKEY does.
Welcome on earth...
A little harsh ;)

The elements you need are selective motion tracking. You run the motion tracking software and eliminate points you don't want. I only know about SynthEyes www.ssontech.com which created a real-time animation by Joe Williamsen (***Warning*** this link contains near nudity Hash A:M clip with near nudity

Then you can blue/green screen as part of the workflow with less issues of lineup.

And I say this in theory as I have never completed that style of workflow.
George Ellis is offline   Reply With Quote
Old January 6th, 2007, 02:29 PM   #9
Inner Circle
 
Join Date: Jun 2003
Location: Toronto, Canada
Posts: 4,750
Quote:
It can be done in a pinch but you would end up rotoscoping the errors so much that you might end up just rotoscoping from the start.
Having done some difference keying, I can tell you that difference keying saves a lot of rotoscoping time!! In my case, the difference key worked well. There were some holes but you can't see it when the clip is playing; a little roto work would fix that (I got lazy). Doing a difference key would cut your roto time to maybe 10-20% it would be otherwise.

2- How well a difference key will work may depend on compression level + camera. I didn't compare different cameras + formats. Motion blur should likely be eliminated as much as possible by increasing the shutter speed.

3- I recall another company trying to do something like Mokey, although I haven't heard about that effort lately. Presumably their automated rotoscoping doesn't work that well.

4- Has anyone tried silhouetteRoto? Does it speed up the roto process a lot?
Glenn Chan is offline   Reply With Quote
Old January 6th, 2007, 03:55 PM   #10
Major Player
 
Join Date: Feb 2004
Location: Philadelphia, PA, USA
Posts: 548
Quote:
Originally Posted by Glenn Chan
4- Has anyone tried silhouetteRoto? Does it speed up the roto process a lot?
Absolutely YES! to both questions.
__________________
Nick Jushchyshyn Matchmoving, Compositing, TD
imdb

Last edited by Nick Jushchyshyn; January 7th, 2007 at 11:08 AM.
Nick Jushchyshyn is offline   Reply With Quote
Old January 6th, 2007, 09:02 PM   #11
Trustee
 
Join Date: Mar 2004
Location: Milwaukee, WI
Posts: 1,719
Quote:
Originally Posted by Glenn Chan
Having done some difference keying, I can tell you that difference keying saves a lot of rotoscoping time!! In my case, the difference key worked well. There were some holes but you can't see it when the clip is playing; a little roto work would fix that (I got lazy). Doing a difference key would cut your roto time to maybe 10-20% it would be otherwise.

2- How well a difference key will work may depend on compression level + camera. I didn't compare different cameras + formats. Motion blur should likely be eliminated as much as possible by increasing the shutter speed.

3- I recall another company trying to do something like Mokey, although I haven't heard about that effort lately. Presumably their automated rotoscoping doesn't work that well.

4- Has anyone tried silhouetteRoto? Does it speed up the roto process a lot?
Yes the difference key can work well but it depends on the type of background and how much overlapping color you have between the subject and the background. I have used it a few times with decent results because there was very little to no similar colors between the subject and the background. It will however not always work as well and you could end up with some really bad results depending on how complex the background is and the color involved. If you can make sure your subject does not share any colors at all with the background then you could get some very clean results.

Also take great care in adjusting the shutter speed or you could end up with elements where the motion doesn't line up and this could give your comp away as fake. You could always use a tool to add back in the motion blur I guess.
Thomas Smet is offline   Reply With Quote
Old January 7th, 2007, 12:26 AM   #12
Regular Crew
 
Join Date: Dec 2003
Location: DC, USA
Posts: 97
Quote:
Originally Posted by Giroud Francois
you described exactly what MOKEY does.
Welcome on earth...
Sorry, I wasn't aware that one must necessarily know the names of professional special effects software packages in order to survive on earth. I thought that's why I stated in the original post, "...and I was wondering if this idea has already been implemented in a software package one can purchase." Keep in mind this forum welcomes hobbyists as well as professionals.

Thanks everyone for the discussion; now I have some things I can look into.
__________________
http://duffx.com
James Duffy is offline   Reply With Quote
Old January 7th, 2007, 07:01 AM   #13
Trustee
 
Join Date: Nov 2005
Location: Sydney Australia
Posts: 1,570
You can build a quite good difference key in Vegas. Your success depends very much on how you shoot it.
In Vegas there isn't a simple difference key FX, rather you need to combine several FXs as well as take care of the compositing.
There's a tutorial here:
http://www.sundancemediagroup.com/ar...sony_vegas.htm

The results aren't spectacular but the footage wasn't shot with the intention of doing a key. With attention to the shooting method and a better camera / less compressed / higher res recording better results would be obtained.
Bob Grant is offline   Reply
Reply

DV Info Net refers all where-to-buy and where-to-rent questions exclusively to these trusted full line dealers and rental houses...

Professional Video
(800) 833-4801
Portland, OR

B&H Photo Video
(866) 521-7381
New York, NY

Z.G.C.
(973) 335-4460
Mountain Lakes, NJ

Abel Cine Tech
(888) 700-4416
N.Y. NY & L.A. CA

Precision Camera
(800) 677-1023
Austin, TX

DV Info Net also encourages you to support local businesses and buy from an authorized dealer in your neighborhood.
  You are here: DV Info Net > The Tools of DV and HD Production > Open DV Discussion

Thread Tools Search this Thread
Search this Thread:

Advanced Search

 



Google
 

All times are GMT -6. The time now is 11:51 AM.


DV Info Net -- Real Names, Real People, Real Info!
1998-2017 The Digital Video Information Network