Name that Artifact! HDV Compression, Tape Head Lag, Sony 30scan, Capture Error

[Matthew Bowman]

I have an artifact showing up throughout HDV 1080p30 and 1080i60 footage shot on the same camera unit. It shows up in two ways. In Final Cut Pro on a 15’’ MacBook Pro at 100% size in the Viewer, the problem shows up as a horizontal bar of distortion (not pixelation) that moves up the frame. It looks like a horizontal bar was cut in the frame and shifted to the left or right and walked up the frame. When viewed at other sizes and after rendering, it looks like motion stutter.

At first, I troubleshot for a mistake during capture to computer or tried to correct the problem through post filters and renders. You can check out my posts on the Apple forums during that process here:

http://discussions.apple.com/thread....9086&#11489086

With no luck there, I’m thinking the problem might have been introduced during acquisition. I’m thinking it’s either an HDV compression blow-out problem or the camera’s tape mechanism is faulty.

I think it may be a compression problem because the issue is more prominent in video with lots of movement and color changes. I think it might be a tape mechanism problem because the distortion artifact covers the frame and moves up in way that loosely resembles a lagging analogue recording.

To be honest, though, these are last resort leads, which I doubt. For one, when playing the source tapes through a deck onto a TV with RCA, I don't see the problem. Then, the artifacts aren’t baked in the capture files and don’t show up in the same place: a clip on loop will show it at different times. That would be the case with compression or tape recording artifacts, yes? Also, the artifact does not resemble the block pixelation or white and black bars that most screen shots of this artifact show. Lastly, the distortion artifact goes away when you freeze a frame.

So, there it is. I have been trying to solve this problem for weeks and would so, so appreciate any ideas.

Here’s a low-res video of example shots with the problem. With the render, it will show up as motion stutter. And remember, they won’t be in the same place - if they show up during the first play - so give it a second watch.

Quicktime file encoded with Motion JPEG B at 720x480 at 25% quality:

http://www.epokhecutmedia.com/video/...nstutter01.mov

The Set-Up and Workflow

Material shot on HDV using the Sony 3-CMOS Z5U in both 1080p30 and 1080i60 in Sony's 30scan setting. The 30scan setting scans a native progressive image at 30fps and splits that into an interlaced frame, recording it as HDV 1080i60. The video was captured from the camera onto a 15'' MacBook Pro with Firewire in both native HDV and ProRes.

[Matthew Bowman]

Where's the pouty lip, tear-in-the-eye emoticon when you need it?

[Chris Soucy]

I'm on a PC (may be no problem) but simply cannot get that clip down. I've left it for half an hour and it still won't finish (yes, I'm on Broadband and it ain't no slouch).

Either chop it or make this a Mac issue per se, as my systems dealings with Quicktime are rather "shas - ta" at the best of times.

Can't diagnose sommat I simply can't see (hey, do you have Dr. House's cell?).

Appologies.


CS

[John Wiley]

Hi Matthew,

Sorry but I also can't get the file to play. Can you put it on vimeo? Or how about some screen grabs?

[Andrew Smith]

I do hope this is regarded as being "helpful". Given the difficulties of downloading the video file per the above posts, I've successfully downloaded it and uploaded it to Vimeo so that we can all see. It's set to only play via embedding in dvinfo.net and no other domain.

Chris, just delete my post if I've overstepped the mark here. Matthew, let me know if you would prefer it to not be on Vimeo and I'll delete it.

Okay ... here goes ...

[edit: clip deleted as Matthew now has his uploaded]

Andrew

[Matthew Bowman]

Sorry for the trouble. The video file is a hefty 112 megabytes, but I didn't think download times would be that bad. I have a horrible, backwoods Brooklyn connection and I can pull it off the server in about three minutes.

Here's the video at Vimeo:

HDV artifact troubleshoot - DVInfo, Apple Forums on Vimeo

Originally Posted by Andrew Smith

Matthew, let me know if you would prefer it to not be on Vimeo and I'll delete it.

Thanks, Andrew, for helping out and putting the video up. It wasn't getting past "We're about to start converting this video" on my system, so I double posted. Could you, though, remove my name from the title? I like to keep search indexes of my name to a minimum.

[Adam Gold]

Matt, I don't see the problem you're describing in the Vimeo clips. And from your troubleshooting and descriptions of the problems, it seems clear the problem is not in within the original tape nor the captured file (if it's in different places each time and goes away when you freeze the frame).

Sounds to me like a computer playback issue. Hardware, software, project settings? If the problem goes away when you freeze the frame I guess you can't just post a frame grab of the problem.

(Also, you can edit your posts so you don't need to double post if you want to make a change.)

EDIT: just went to the apple forums and read your thread there. I think you may have the wrong expectations, when you say "but the video still jitters badly during slow pans" in 30p. Well, yes. It's supposed to. That's what 30p is all about. It will never be as smooth as 60, i or p.

Without knowing all the details about your hardware and seeing the problem, it's tough to help. Based on the limited HW info you posted on the other forum, it might also be a HDD problem (you mentioned one of your disks is 5400rpm... not good). I'm not a Mac person so I don't know what "with all the fixin's" means... chip, speed, RAM, video card? All these matter and it could be you're just short on horsepower for editing HD.

Wish I could help more.

Edit 2: Just did a little more research on the Apple website to try to figure out potential hardware issues (why, I'm not sure; I really should be editing). Now I could be wrong about this, but even the newest 15" MBPs offer three different chips, the best of which is about half the horsepower of the minimum chip you'd put in a decent desktop PC for HDV editing. It's woefully low on RAM, the video card isn't even close to what you'd need for proper playback, and as mentioned earlier only comes with a 5400RPM HDD, which isn't enough for HD in any form. If it's an older MBP (Pre-April 2010) then the specs would be even lower -- as low as 1/4 the chip speed and 1/8 the RAM of the latest model).

So to me the Hardware is the obvious weak link. But as I said, I could be wrong (and have been many times before). Alternatively, could easily have something to do with your project setup or settings in FCP, about which I know nothing and am therefore useless. But because these artifacts do not appear on my system with your clips, I'm still leaning towards hardware.

[John Wiley]

Hi Matthew,

My first impression from your description was also that it was a playback issue rather than something wrong with the original clips. The fact that we can't notice anything on vimeo would support this.

Without knowing your exact setup it is hard to know what the problem is, but if you can be sure that your footage (original and edited/exported) is free from atrifacting when played on a DVD player/another computer then you may just have to put up with a few out of place blocks in your preview window.

[Matthew Bowman]

First,

Thank you everyone very, very much for your posts. Adam, you went above and beyond, and for that - whether your efforts were a needed distraction from work or not - you're awesome.

Now,

You guys aren't seeing the motion stutter! I'm floored. I had checked my video on a few systems and always saw the stutter. Every system was prosumer horsepower, though, and while I thought they could handle it, maybe not. To be honest, my hardware knowledge sadly trails my attention to software.

Even though so much has pointed to playback, I did not think the issue was weak hardware because the stutter shows up for me even in low-res renders. Whatever the source, this made me think the stutter was being baked into my renders.

For example, shouldn't my computer be able to play Photo JPEG video at 25% quality without hanging up!? I play other HD videos off Vimeo just fine. Why would my video be lagging?

For this reason and since the stutters happens at different times during playback, I was starting to think my computer was making some sort of compression error.

Here's a new observation, though: I just noticed the sample video's timecode, generated from a filter after capture, skips along with the rest of the video, meaning the motion lag isn't from capture!

Before I go on, here's my hardware specs:

-----Computer: 15" MacBook Pro
-----Processor: 3.06 GHz Intel Core 2 Duo
-----RAM: 8 GB 1067 MHz DDR3
-----Local HDD: 500 GB 7200rpm, split into 115GB Boot and 385 Media partitions
-----External FW800 / USB HDD's: 1TB 5400rpm G-RAID mini, 2TB 7200rpm Seagate
-----Graphics Card: NVidia GeForce 9600M GT w/ 512 MB VRAM
-----HDV Deck / Camera: Sony Z5U w/ FW400 port to FW800 MBP port.

For capture, I'm stuck with only one Firewire port and no eSATA or PCI bus. So, I have tried capturing:

-----(1) from my camera using FW400 to 800 to my local HDD
-----(2) from my camera using FW400 to 800 to an external 7200 HDD using USB
-----(3) from my camera using USB to an external 7200 HDD using FW800

For playback, then, my source files have been on both my local drive and my external 7200 drive with FW800.

If you'll allow this bravado, I don't think the issue is from project or sequence settings. I know Final Cut Pro well and have tried every single ridiculously remote setting and possibility to fix this - during capture, after capture, and with renders - with no success.

[Andrew Smith]

From your original description, I was wondering if your video card was having difficulty keeping up with displaying the video at the size that it was at on your screen.

In any case, if it's not showing up on a final export from your workflow then it's not a deal-breaker with regards to getting work done and out the door.

Andrew

[Adam Gold]

I think Andrew is right; it's annoying but not really affecting your output.

A closer look at your specs reveals that things are worse than I had feared: while you have plenty of RAM, your chip is even slower than I suspected, on the order of having about 40% of the required speed/power necessary for HDV. Imagine knowing you needed a 300hp car to get up a hill, and trying to do so with a 180hp engine. You might get to the top eventually, but not quickly and smoothly.

And partitioning your internal HDD is always a lousy idea for this purpose: by having your media on this physical disk as well as your OS, Programs and Project, you are mercilessly banging away at the same disk, which will surely cause a bottleneck in playback. Rendering, all it does is slow you down because it doesn't have to happen in real time, but it really shows up in the compromised playback you are experiencing. And you don't say how the external drives are actually connected, FW or USB (can't be both). USB is too slow to edit from. And while most NLEs don't actually use the video card for editing purposes, they do for preview and yours appears to be quite underpowered.

Imagine that in our car scenario above -- and I know I'm stretching the analogy to the breaking point, but bear with me -- the manual specifies premium fuel at 93 octane, but you fill the car with the cheapest 87 octane gas you can find. Even if the car drives it'll bang and ping and probably blow itself apart if you push it too hard.

It's also possible that the Photo JPEG format is causing issues; I don't believe it's designed for normal HD Video playback, but I could be wrong about this. I've never even considered using Photo JPEG for anything so I'm the wrong guy to comment. You may have good reasons for doing this, but I'd examine this carefully and try a more conventional codec if possible. My understanding of FCP has always been that it's best to let it convert your HDV footage to AIC and edit that. But my knowledge of FCP is sketchy at best.

This is clearly not a capture issue and the problem is not with your captured files. And motion stutter, yes, we see that, but that's because it's 30p. But no unexpected artifacting showing any kind of defect, not the way you described it in your first post, no. If you want smooth pans, shoot 60i.

The real way to test this is to capture some footage to your Mac from the Z5 and burn a DVD of it. Then A/B switch it by playing on a real TV set from both the DVD and straight from the cam. Aside from the resolution difference (if you go to an HDTV via HDMI from the cam vs. regular DVD) all should be the same.

Also note that you can't capture from the Z5 using USB; you must use FW. And although you may have a FW800 port on the Mac, the cam can only output FW400, no matter what kind of cable you use.

So I think Andrew has nailed it: you either live with this or upgrade your hardware. Most of the other posts on this board advise against trying to edit HDV or especially AVCHD on a laptop; all except the most expensive gaming rigs are insufficient for this purpose. The cheapest Mac Pro Tower I priced out today that would reliably handle HDV was $7,000.00. I couldn't find any way to configure an MBP to include the specs I'd be comfortable with. SD DV with iMovie, sure. HDV with FCP, not so much.

[Matthew Bowman]

It's for sure. It's playback.

I did one of the few things I hadn't tried in order to check playback. I printed to video (printed back to miniDV) and played the clips on a TV from my camcorder. No problems.

I had told myself, "It's not playback! You've checked your video on other, heavier systems and you can't play low-res renders!" So I never checked for sure by printing out again to tape.

What's really bugging me, though, is that I have the most beefed up 15'' MacBook Pro you can buy. Shouldn't I be able to play Photo JPEGs at 25% quality. Photo JPEG is a Mac recommended offline format. With the HDV long GOP removed and the quality so reduced, I thought for sure the lags couldn't be from hardware speed. But I'll play with this and see what I can learn.

I'd like to hear if you guys have any favorite Mac system monitoring software for watching for bottlenecks. I always keep Activity Monitor running, but I wonder if there is better software.

Thanks again for all your guys' help. It's very, very nice to know my video is fine and I can move forward. If you knew the hours I've burned working through this....

Adam, thanks again for the lengthy posts. You've given me some new stuff to think about. For the time being, I need to learn about my computer's limits and better tailor my system, workflows, and settings. I appreciate the time you spent researching and responding.

So.... Cheers to DVInfo! I'm off to get the rest of video out the door - finally. It feels good.

[Adam Gold]

Thanks for coming back and closing the loop on this. You'd be amazed at how many people don't.

And hats off to you for sticking with this... I'm amazed you're able to get anything done at all with that hardware.... ;-).

[Andrew Smith]

Hey Matthew, it's great to see that you got a solution on that one ... and much faster than that "other" discussion forum. :-)

Ditto-thanks for updating us. Drop by any time.

Andrew

[Dale Guthormsen]

Mat,


Do not feel like the lone star ranger here!!!

I have seen this from my workstation!!! Caused me great anxiety. I kind of felt it was a graphics card issue as I had just switched to Edius 5.0 on an I7 procesor and 10 gigs of ram!! Sense it does not show up when I burn to blu-ray I just have kind of let it go. and do not really concern myself with it. I also thought it might be the monitor.

did you actually find a cure?


Dale Guthormsen