Rotoscoping software idea -- does this exist? at DVinfo.net

James Duffy · January 5th, 2007, 12:40 PM

I had an idea which would make certain rotoscoping so much simpler, and I was wondering if this idea has already been implemented in a software package one can purchase.

The idea is that the camera sits on a tripod, motionless, and films the background without the subject. The subject is then filmed in front of the background doing whatever is necessary for the take. The software would then compare each frame which contains the subject with the footage that does not contain the subject, and through a series of tests such as examining the color differences between the frames, would be able to isolate the subject from the background. This would allow one to use green screen effects without a green screen, do digital DOF effects (applying a gaussian blur to the background but leave the subject unchanged), etc.

I have no idea how well it would work in practice, but it doesn't seem like it would be difficult to implement. Has this been done, and if not, is there any reason why it wouldn't work?

Nick Jushchyshyn · January 5th, 2007, 01:08 PM

Yes it exists ... it's a difference key operation in most compositing apps (and some NLE programs) ... and it typically doesn't work very well.

Reasons why it doesn't work include noise (every pixel is a little different, all the time) shadows (it's still the background, but it's different than before) and color similarities between foreground and background (if you have a scene that calls for everything to be lit with an eery blue light ... forground and background will have the same color pallet)

James Duffy · January 5th, 2007, 01:38 PM

I was thinking something more advanced than a single operation, like subtracting one image from another. That's why I was thinking you film the background without the subject and not just have a screen grab of it, so the software could account for the camera's natural grain. Color difference alone wouldn't account for it, since skin tone could match shades of the background, but I was thinking a more advanced algorithm that used a combination of tests for edge detection to identify moving objects, and show only the color information from these moving objects.

Again, not sure if it would work properly, just an idea. If you close one eye, you can still identify a clear boundary where one object ends and another begins. Of course, maybe to work properly it would require significant advances in artificial intelligence and computer vision.

Nick Jushchyshyn · January 5th, 2007, 01:52 PM

Mmmm ... that sounds like optical flow...
http://www.fxguide.com/article333.html

James Duffy · January 5th, 2007, 02:29 PM

Hmm, optical flow looks to me to be a combination of Realsmart Twixtor, 2d3 Steadymove, and WinMorph, although probably more powerful/flexible. I'm not familiar enough with the technology behind it to know if it could be adapted to extract moving objects from a still background.

I did find this: http://www.dvxuser.com/V6/showthread.php?t=59238
He seems to be detailing the same thing I'm describing, but using an advanced technology in-camera to assign a z-index (depth) to each pixel, which could then be used in post to apply effects to objects within depth-ranges. That would be amazing, and much cleaner than what I'm suggesting, but it would also require new technology to be doable. I'm wondering if it's possible to do with current video technology through motion detection of 2-d images.

Well either way, I can't wait for the day when the technology behind his idea is implemented.

Thomas Smet · January 5th, 2007, 04:15 PM

The grain however wouldn't match up. If noise was constant it may have a fighting chance but noise changes all the time. That means it would be almost impossible to match up the same noise pattern between the foreground plate and the background plate.

The other reason why this doesn't work very well is because people are not soilid color objects and chances are that a few or more pixels will match the color of the background plate. This would mean you would end up with holes in your subject. This is why blue and green screens are used because it is easy to setup the subject so they do not share any of the same chroma information as the screen.

A keyer of this type would have to have a butt load of error correction built in to try and compensate for the subject being over similar colors.

For example I mean lets say your subjects is wearing a brown coat and they walk in front of a brown fence. Maybe not at all of brown pixels will give you trouble but some of them will.

It can be done in a pinch but you would end up rotoscoping the errors so much that you might end up just rotoscoping from the start.

This method has been used in the past with keyers such as Ultimate. There was a screen correction filter that was used to clean up badly lit bluescreens. What you would do shoot a clean plate of the bluescreen and then shoot your subject in front of the bluescreen. You then used the filter to clean up the quality of the bluescreen. This was not a keyer on it's own but just a way to help get a cleaner key.

Giroud Francois · January 6th, 2007, 03:45 AM

you described exactly what MOKEY does.
Welcome on earth...

George Ellis · January 6th, 2007, 10:44 AM

Quote:

Originally Posted by Giroud Francois

you described exactly what MOKEY does.
Welcome on earth...

A little harsh ;)

The elements you need are selective motion tracking. You run the motion tracking software and eliminate points you don't want. I only know about SynthEyes www.ssontech.com which created a real-time animation by Joe Williamsen (***Warning*** this link contains near nudity Hash A:M clip with near nudity

Then you can blue/green screen as part of the workflow with less issues of lineup.

And I say this in theory as I have never completed that style of workflow.

Glenn Chan · January 6th, 2007, 02:29 PM

Quote:

It can be done in a pinch but you would end up rotoscoping the errors so much that you might end up just rotoscoping from the start.

Having done some difference keying, I can tell you that difference keying saves a lot of rotoscoping time!! In my case, the difference key worked well. There were some holes but you can't see it when the clip is playing; a little roto work would fix that (I got lazy). Doing a difference key would cut your roto time to maybe 10-20% it would be otherwise.

2- How well a difference key will work may depend on compression level + camera. I didn't compare different cameras + formats. Motion blur should likely be eliminated as much as possible by increasing the shutter speed.

3- I recall another company trying to do something like Mokey, although I haven't heard about that effort lately. Presumably their automated rotoscoping doesn't work that well.

4- Has anyone tried silhouetteRoto? Does it speed up the roto process a lot?

Nick Jushchyshyn · January 6th, 2007, 03:55 PM

Quote:

Originally Posted by Glenn Chan

4- Has anyone tried silhouetteRoto? Does it speed up the roto process a lot?

Absolutely YES! to both questions.

Thomas Smet · January 6th, 2007, 09:02 PM

Quote:

Originally Posted by Glenn Chan

Having done some difference keying, I can tell you that difference keying saves a lot of rotoscoping time!! In my case, the difference key worked well. There were some holes but you can't see it when the clip is playing; a little roto work would fix that (I got lazy). Doing a difference key would cut your roto time to maybe 10-20% it would be otherwise.

2- How well a difference key will work may depend on compression level + camera. I didn't compare different cameras + formats. Motion blur should likely be eliminated as much as possible by increasing the shutter speed.

3- I recall another company trying to do something like Mokey, although I haven't heard about that effort lately. Presumably their automated rotoscoping doesn't work that well.

4- Has anyone tried silhouetteRoto? Does it speed up the roto process a lot?

Yes the difference key can work well but it depends on the type of background and how much overlapping color you have between the subject and the background. I have used it a few times with decent results because there was very little to no similar colors between the subject and the background. It will however not always work as well and you could end up with some really bad results depending on how complex the background is and the color involved. If you can make sure your subject does not share any colors at all with the background then you could get some very clean results.

Also take great care in adjusting the shutter speed or you could end up with elements where the motion doesn't line up and this could give your comp away as fake. You could always use a tool to add back in the motion blur I guess.

James Duffy · January 7th, 2007, 12:26 AM

Quote:

Originally Posted by Giroud Francois

you described exactly what MOKEY does.
Welcome on earth...

Sorry, I wasn't aware that one must necessarily know the names of professional special effects software packages in order to survive on earth. I thought that's why I stated in the original post, "...and I was wondering if this idea has already been implemented in a software package one can purchase." Keep in mind this forum welcomes hobbyists as well as professionals.

Thanks everyone for the discussion; now I have some things I can look into.

Bob Grant · January 7th, 2007, 07:01 AM

You can build a quite good difference key in Vegas. Your success depends very much on how you shoot it.
In Vegas there isn't a simple difference key FX, rather you need to combine several FXs as well as take care of the compositing.
There's a tutorial here:
http://www.sundancemediagroup.com/ar...sony_vegas.htm

The results aren't spectacular but the footage wasn't shot with the intention of doing a key. With attention to the shooting method and a better camera / less compressed / higher res recording better results would be obtained.

January 5th, 2007, 12:40 PM	#1
James Duffy Regular Crew Join Date: Dec 2003 Location: DC, USA Posts: 97	Rotoscoping software idea -- does this exist? I had an idea which would make certain rotoscoping so much simpler, and I was wondering if this idea has already been implemented in a software package one can purchase. The idea is that the camera sits on a tripod, motionless, and films the background without the subject. The subject is then filmed in front of the background doing whatever is necessary for the take. The software would then compare each frame which contains the subject with the footage that does not contain the subject, and through a series of tests such as examining the color differences between the frames, would be able to isolate the subject from the background. This would allow one to use green screen effects without a green screen, do digital DOF effects (applying a gaussian blur to the background but leave the subject unchanged), etc. I have no idea how well it would work in practice, but it doesn't seem like it would be difficult to implement. Has this been done, and if not, is there any reason why it wouldn't work? __________________ http://duffx.com

January 5th, 2007, 01:08 PM	#2
Nick Jushchyshyn Major Player Join Date: Feb 2004 Location: Philadelphia, PA, USA Posts: 548	Yes it exists ... it's a difference key operation in most compositing apps (and some NLE programs) ... and it typically doesn't work very well. Reasons why it doesn't work include noise (every pixel is a little different, all the time) shadows (it's still the background, but it's different than before) and color similarities between foreground and background (if you have a scene that calls for everything to be lit with an eery blue light ... forground and background will have the same color pallet) __________________ Nick Jushchyshyn Matchmoving, Compositing, TD imdb

January 5th, 2007, 01:38 PM	#3
James Duffy Regular Crew Join Date: Dec 2003 Location: DC, USA Posts: 97	I was thinking something more advanced than a single operation, like subtracting one image from another. That's why I was thinking you film the background without the subject and not just have a screen grab of it, so the software could account for the camera's natural grain. Color difference alone wouldn't account for it, since skin tone could match shades of the background, but I was thinking a more advanced algorithm that used a combination of tests for edge detection to identify moving objects, and show only the color information from these moving objects. Again, not sure if it would work properly, just an idea. If you close one eye, you can still identify a clear boundary where one object ends and another begins. Of course, maybe to work properly it would require significant advances in artificial intelligence and computer vision. __________________ http://duffx.com

January 5th, 2007, 01:52 PM	#4
Nick Jushchyshyn Major Player Join Date: Feb 2004 Location: Philadelphia, PA, USA Posts: 548	Mmmm ... that sounds like optical flow... http://www.fxguide.com/article333.html __________________ Nick Jushchyshyn Matchmoving, Compositing, TD imdb

January 5th, 2007, 02:29 PM	#5
James Duffy Regular Crew Join Date: Dec 2003 Location: DC, USA Posts: 97	Hmm, optical flow looks to me to be a combination of Realsmart Twixtor, 2d3 Steadymove, and WinMorph, although probably more powerful/flexible. I'm not familiar enough with the technology behind it to know if it could be adapted to extract moving objects from a still background. I did find this: http://www.dvxuser.com/V6/showthread.php?t=59238 He seems to be detailing the same thing I'm describing, but using an advanced technology in-camera to assign a z-index (depth) to each pixel, which could then be used in post to apply effects to objects within depth-ranges. That would be amazing, and much cleaner than what I'm suggesting, but it would also require new technology to be doable. I'm wondering if it's possible to do with current video technology through motion detection of 2-d images. Well either way, I can't wait for the day when the technology behind his idea is implemented. __________________ http://duffx.com

January 5th, 2007, 04:15 PM	#6
Thomas Smet Trustee Join Date: Mar 2004 Location: Milwaukee, WI Posts: 1,719	The grain however wouldn't match up. If noise was constant it may have a fighting chance but noise changes all the time. That means it would be almost impossible to match up the same noise pattern between the foreground plate and the background plate. The other reason why this doesn't work very well is because people are not soilid color objects and chances are that a few or more pixels will match the color of the background plate. This would mean you would end up with holes in your subject. This is why blue and green screens are used because it is easy to setup the subject so they do not share any of the same chroma information as the screen. A keyer of this type would have to have a butt load of error correction built in to try and compensate for the subject being over similar colors. For example I mean lets say your subjects is wearing a brown coat and they walk in front of a brown fence. Maybe not at all of brown pixels will give you trouble but some of them will. It can be done in a pinch but you would end up rotoscoping the errors so much that you might end up just rotoscoping from the start. This method has been used in the past with keyers such as Ultimate. There was a screen correction filter that was used to clean up badly lit bluescreens. What you would do shoot a clean plate of the bluescreen and then shoot your subject in front of the bluescreen. You then used the filter to clean up the quality of the bluescreen. This was not a keyer on it's own but just a way to help get a cleaner key.

January 6th, 2007, 03:45 AM	#7
Giroud Francois Inner Circle Join Date: Feb 2004 Location: switzerland Posts: 2,133	you described exactly what MOKEY does. Welcome on earth...

January 7th, 2007, 07:01 AM	#13
Bob Grant Trustee Join Date: Nov 2005 Location: Sydney Australia Posts: 1,570	You can build a quite good difference key in Vegas. Your success depends very much on how you shoot it. In Vegas there isn't a simple difference key FX, rather you need to combine several FXs as well as take care of the compositing. There's a tutorial here: http://www.sundancemediagroup.com/ar...sony_vegas.htm The results aren't spectacular but the footage wasn't shot with the intention of doing a key. With attention to the shooting method and a better camera / less compressed / higher res recording better results would be obtained.