Multi Core Rendering

[Ryan Hansen]

Well, Here we go again, from what I've seen in the world everyone has issues with this sorta stuff and FCS. Im wanting to render some motion graphics throught compressor, and thats fine, but when i send it to my cluster it still only uses 2 cores and not even 100% (averaging maybe 60-70%

I found a number of tutorials for multicore rendering, but I can't get it to work with me.... Here are the details:

MacPro 2.8ghz 8 core (2006 or 7 I think)
4gb Ram
FCS3
OS X 10.6
Separate OS X and Scratch disks

Attached are my screenshots, hopefully someone will know how to fix this, or if I need to upgrade anything - Ram is already on my list of things to do....

While I'm here, every time I open Q-Master in the sys prefs it has to close and open again. Normal? Bug? Just Me?


Thanks
Ryan

[Robert Lane]

Simple fix:

There are 2 issues here, one is the setup:

You've setup your cluster to work as if you're spreading the rendering jobs across a network allowing other computers to share your resources, that's not what you're after. You don't want to *share* you want to *Manage*.

Uncheck Share and check Manage instead, then under "Options for selected service" make sure Compressor is selected in the list above then hit the "options..." button and select up to *half* of the available cores on your machine. Then when you submit your job to Compressor you check the box "include unmanaged services".

BUT, only MPEG-2 encodes will spread across multiple cores, that's because MPEG-2 processing is by default multi-threaded. AVCHD and H.264 are NOT multi-threaded codecs so even with distributed processing turned on you will only use one of your CPU cores, period. Unless someone creates a multi-threaded version of those codecs that behavior will never change. So in fact what you are seeing in your Activity Monitor is correct: the H.264 encodes are only using a single core. There's nothing more you can do in Compressor or Qmaster to speed that up unfortunately.

That is one reason Matrox and BlackMagic have created hardware cards that accelerate H.264 encoding because you can't do it in software. If you're planning on working with H.264 encodes a lot it would probably be beneficial for you to check into the hardware encoding options.

For details on this setup pick up the book from Brian Gary, Compressor 3 Quick Reference Guide.

[William Hohauser]

If you are using Snow Leopard, the QMaster sys pref window will open/close/reopen because that particular system pref is still in 32-bit code!

I just set up a new 8-core and had a question about Qmaster, Final Cut and multi-processing. When I render in FCP, usually four cores activate to about 60 to 75% while the other cores usually remain below 5% as if they are being used for background computer processes. Sometimes a fifth or sixth core pops up for a few seconds. Is that a limitation of the way FCP renders or the fact that I only have the stock 6gb of RAM installed? In the searches I've done in the Apple literature, Qmaster is never associated with FCP only Compressor. Have I missed something here?

[Robert Lane]

That's right, Qmaster is for sharing resources for Compressor encoding only, the idea being that you could turn multiple Mac Pro's into a render-farm by distributing the workload across dozens of cores to finish a large-scale job. However the promise of Snow Leopard being able to boost render performance in FCP7 only occurs with newer GPU cards that are actually designed to leverage OpenCL technology.

Only mainframe style systems such as Quantel allow shared processing for rendering on the timeline because each workstation is literally being fed it's assets via fiber from a huge rack of data servers with TB's of RAM and massive processing power behind them.

[William Hohauser]

Does GPU performance come up in the Activity monitor graphs?

[Ryan Hansen]

Originally Posted by Robert Lane

Simple fix:

You've setup your cluster to work as if you're spreading the rendering jobs across a network allowing other computers to share your resources, that's not what you're after. You don't want to *share* you want to *Manage*.

Duh.... I should have seen that... Done it now :) Thank you for your help.

Interesting what you say about the H.264 codec only being able to encode on 1 core, I would almost expect someone to come up with one one day since everything is heading multi-core.... I was always under the impression that the software / hardware did all the work and that the codec was just the "settings" and specs that the software / hardware was working to..... I'm Guessing that its something to do with the way the codec processes the files - I've heard that some codecs do some funky stuff with the order of frames and sync with the audio and all sorts of stuff....

[Robert Lane]

Originally Posted by William Hohauser

Does GPU performance come up in the Activity monitor graphs?

No the current AM in 10.6 only shows CPU, RAM and Disk monitors. There may be another utility that talks to OpenCL threads but I haven't seen one yet. But that also would require one of the newer ATI or nVidia cards that has GPU firmware to *talk* to OpenCL for it to do it's offloading.

What's interesting - and something that I can't speak to directly as to "why" - is that Premiere Pro, Avid and Media 100 already use built-in software threading to leverage any GPU the system is connected to (assuming the card has 256MB or more of VRAM). Edius does this as well but that's a PC-only app. Why FCP7 will only do this under OpenCL and only with these specific cards is a total mystery. Maybe the spec will open up as FCP7 matures and more GPU's become available.

This retail white-paper enclosed on OpenCL gives little insight into the how and why of the card-specific code and stuff on ADC is technically beyond my understanding since I'm not a code monkey, but it's definitely GPU-firmware specific; if the card wasn't written for it then OpenCL get's unused. Which is a real shame considering people who just purchased Mac Pro's prior to Snow Leopard and opted for the more expensive cards can't use OpenCL. It's another head-banger from Apple.

[Ryan Hansen]

Does anyone know of any 3rd party apps that monitor GPU? or even a llist of supported hardware? Ive got a nVidia 8800gt, but I can't find any definite answers....

[Robert Lane]

Originally Posted by Ryan Hansen

Does anyone know of any 3rd party apps that monitor GPU? or even a llist of supported hardware? Ive got a nVidia 8800gt, but I can't find any definite answers....

You don't need any third-party apps, take a look at this thread and use the Developer Tools built into OSX:

http://www.dvinfo.net/forum/non-line...rformance.html

[Ryan McBride]

Ive been trying to set up my Mac Pro QMaster this way. I found this article that explains it with the share option selected. Any idea why?

Compressor - Enabling multiple Cores

[Ryan Hansen]

Yea that's the one I went off, I think shared is more for network rendering

Alot of good stuff on that site as well :)

[Ryan McBride]

Thats what I thought. I also came across this thread that shows a slightly different set up as well. Using Compressor with multiple cores - Digital Rebellion Blog

[William Hohauser]

Reopening this thread for some more questions and observations.

1) Setting my 2.66 eight-core to 4 instances in Qmaster works great. All cores light up during MPEG2 and even h.264 rendering in Compressor. Actually the Activity Monitor shows 16 cores. However once the task shifts to ac3 audio files it drops down to 1 to 3 cores at 40% at most. Anyway, 1 hour of DV quality material took 25 minutes to render at the default high quality 2-pass DVD setting Apple provides. Setting Qmaster to anything over 4 instances didn't work and the computer would use the default render settings as if the Quickcluster hadn't been set up.

2) Sending a sequence with real-time (dark green "Full") effects to the Quickcluster through Compressor causes an immediate render failure although sending the same sequence to render under the "My Computer" option works perfectly. Creating a reference quicktime and bringing that in Compressor fixes this problem since the real-time areas are rendered at this point. The Share function has the same problem unless you render everything before choosing the Quickcluster. This seems to be a programming mistake that should be corrected.

[Ryan Hansen]

I take it your using 10.6 nd fcs3?

[William Hohauser]

Yes, the computer and programs are completely updated.... except for the Pro Apps update that just showed up today.

The end effect, even with the rendering issues, is that I used to run a real-time dub to a DVD recorder to save time on certain projects. Now I can make DVDs for clients in the computer in less time.