System for VFX
 

Hi
I need to build a system solution for visual effects. I'll need two workstations and a render farm for a particular project to handle 2K uncompressed DPX/TIFF files and output to same. I'll need as close to real time performance as I can, but it's not critical. First workstation will run Nuke and After Effects and be a compositing/graphics app. The second one will run Maya and be a 3D Animation station.

Will I really need a dual Xeon, 24GB RAM, RAID 0, SSD and Nvidia Quadro beast for both the workstations, or can I get by with a 'smaller animal' and focus on the render farm? Here's how I see the options:

1. Full spec workstations and farm
2. Full spec WS and a 'made-with-old-computer-parts' farm
3. Less powerful WS and full spec farm

Am I totally wrong in going about things this way? Of course, I wouldn't be asking these questions if price wasn't a factor. At the most I might double the number or workstation for each app, but the project doesn't need more than four artists. Any help will be appreciated. Thanks in advance.

no answer?
 

Surely someone must be running a compositing suite here...

Can you provide more info:

1) Which app(s) do you need the render farm for - Nuke (5, 6 or 6x), AE (CS4 or CS5) and/or Maya (2010 or 2011)? If Maya, will you be using an external renderer such as RenderMan or V-ray?

2) What needs to be 'real-time' - compositing or rendering and which programs?

3) What is your budget?

I have a dual 6-core Xeon PC (HP Z800) with 24GB of ram & 6-drive 12TB Raid 5 array as my work PC and use NukeX 6, AE CS5 and Cinema 4D. For my personal PC, it only has a single quad i7 with 12GB of ram and a similar 12TB Raid 5 array. Rendering and previewing is sooo much quicker on the HP Z800. It only takes 3-5 minutes for a single frame to render in C4D on the Z800 versus 8-15 minutes on my personal PC.

With the added cost of render licenses, spending extra for a very fast workstation rather than cheaper & slower PCs can help offset the additional cost.

FYI, for render computers, it is best to have a minimum of 1GB ram per core and 2GB per core is optimal.

For uncompressed 2K editing, you need a direct attached Raid system capable of 500+ MB/s on average for 2 streams. I prefer an Areca 1880x and a PROAVIO/Enhance Tech 8 bay SAS/SATA box and 8 1TB Samsung F3 or Seagate 7200.12 drives in either Raid 0 or Raid 5.

Thank you for replying. To answer your questions:

1) I need to render from Nuke and AE CS5, but never at the same time.

2) Compositing needs to be as real-time as possible. I'm not doing Transformers but it's still film work, and creative decisions will probably be changed with every iteration. I really don't know where my bottle-neck will be.

3) What is your budget?
As cheap as possible. I don't mean this in a sarcastic way at all. I just don't want to go for ready solutions like Boxx, HP, etc because I'm comfortable with computer hardware. I just want to go to a computer store and pick up what I want. If something burns, I'll just repeat the process. It'll still be cheaper in the long run. The downtime in this case is acceptable.

Thank you for letting me know what you use. I cannot tell you how much that has helped. Since I've posted this, I've learnt the CGI/3D work on this project is minimal, and I can outsource it to a freelancer. So the work is now purely compositing and motion graphics. I'm hoping my DI facility will accept linear TIFF files!

I would really appreciate and be grateful for any additional insight you can provide. Thanks.

Check out the eVGA SR2 motherboard which allows overclocking the latest Xeon CPUs. To save money, you could get the E5540 (2.4GHz) 4-core and raise to above 3GHz or for a seriously fast system, go with the 6 core X5650 (2.66GHz) and OC to 3.6GHz+. Either way, I'd go with 24GB ram (6 x4GB sticks).

Thanks Steve.

Here's what I've come up with for the main Workstation:
Six core i7 and 24GB RAM (I spoke to a lot of Nuke users and they told me the i7 was 'good enough' for my specific purposes)

OCZ SSD Boot and OS Drive (120GB)
OCZ X2 PCI-E Drive (320GB) - Cache (650+ MB/s)
OCZ X2 PCI-E Drive (320GB) - Read
3x1TB 7200rpm Drives for backup and write (WD)

Nvidia GTX 580 - direct out to a 27" Dell with a Sypder or Eye-one calibration kit.
The GPUs directly output RGB linear in 2K and I'll be working in sRGB linear. The Dell can be simulated to look 'something like' a film out just to make sure the comps don't fall apart. If I can't correct the GPU gamma I might go with a AJA or BM and might add a second monitor too.

Write out to DPX log and back to the DI!

For me specifically, the major problem is getting the OCZ drive in India. It's not available from their authorized dealers and I'll probably have to import them (adding 30% to the cost).

Thanks again!

I don't see the benefit of spending so much money on the OCZ X2 drives.

About the other Nuke users: no disrespect to any of them, but I bet very few have a good understanding of computers. I would classify most Nuke users as 'Artists', not computer nerds (like me). If they are working for a company, chances are that they have no involvement in the hardware they use.
...Just something to consider....

About the LCD: I have read about new NEC 10 bit LCDs, which are cheaper than the Eizo but still provide a far better display than any Dell. I know NEC has a 24" and a 30" 10bit LCD, and I vaguely recall a 27" (2560x1440) as well for nearly the same price as the 24" Eizo. I believe NEC sells them with and without their calibration device.

Maybe the Dell will work 'good enough' with a lot of fanagling with profiles, simulated color spaces and LUTs. However, a Nuke artist really deserves better than 'good enough'.

I see absolutely no benefit using the GTX 580 seeing as Nuke & AE do not fully leverage the GPU aside from a few plug-ins (including a tracker from The Foundry). For roughly $200 more than the 580, I would rather use the Quadro 4000 for its 10bit output. If I am missing something, please correct me.

What do you mean by "Backup and write" for the 3 drives? And how would you setup each drive - separately or in Raid 0?

FYI, if you ever want to use a drive in Raid, DO NOT use Western Digital's Caviar (Blue or Black) drives because WD crippled them, which causes them to 'drop' out of Raid arrays; thus, the array is either degraded or completely lost. WD did this to force consumers into the RE4 enterprise drives. Personally, I prefer Seagate and have not had a single dead drive from them in the last 4 years. I have 16 2TB Seagates (6 Barracuda XT, 6 Constellation ES, 4 5900rpm for backup) and 12 1TB 7200.12 drives between my work PC and home PC (in addition to several full Hitachi 2TB as backups).

Furthermore, I have 3 Intel X25 80GB SSD drives as OS drives in my 2 PCs and a laptop.

On a side note: have you ever been a 'post grad' at FXPHD.com? This is where I learned to use Nuke, advanced AE and Cinema 4D. If not, where did you go to learn Nuke?

The X2 gives spectacular read and write speeds - and it's not very expensive. I could string up a few RAIDS, but why bother, when even at RAID 0, I'll still struggle to get uncompressed 2K through the pipeline? Anyway, I might have to RAID0 two SSDs (as the cache drive) if I'm unable to purchase the X2.

I've used the nuke email forum (foundry), dvinfo, cow, vfxtalk and the nuke's user guide to make these decisions. This is what the manual spells out clearly:
1. Nuke supports multiple processors - the more/faster the better
2. Nuke loves RAM - the more/faster the better
3. Nuke desperately needs the fastest cache drive possible
4. Nuke really (and desperately) needs the fastest read drive possible. The OS drive isn't an issue.
5. Nuke supports OpenGL (over any other technology) and recommends Quadros. The viewer needs to be specifically told to use the GPU for output if needed.

Why did I choose one i7 over 2 Xeons? Most of the issues with uncompressed footage are due to slow drives and incorrect system configurations. The CPUs are never fully utilized, except when everything else is perfect. I saved on the CPU and put that money into the SSDs. It's a trade-off based on this specific project.

LCD - After having read the Color, Da vinci and Lustre manuals, and at least ten other white papers on grading, I've learnt (as if I had another choice) that only expensive projectors can emulate the film environment. Also, the size of the screen is critical in observing tonality, among other things. The best I can do, is working in sRGB and carry out a few tests with the DI facility beforehand. Most of the compositors in the nuke forum work on the dell 27" and rave about it. The Eizos, dreamcolors are only marginally better, so why pay more? I will be calibrating and testing with the DI facility anyway. If the monitor doesn't work, I'll buy an Eizo or an HP (Eizo has no service in India, Dell and HP do).

About GTX vs FX - this is where things are still a bit murky. Nvidia has a separate SDI output card that needs to be connected to their quadros - for monitoring work. They don't recommend connecting the 10-bit quadros directly to a monitor (But it's okay for gamers playing in 2K!). I have a feeling it's something to do with the gamma that these cards output to (the output is 10-bit RGB linear), but I couldn't find a single thread or paper that tells me why I really need an I/O card - whether Matrox/BM/Aja or Nvidia. Fishy.
The GTX 580 can drive two 2K monitors.

The three 1TB drives are for writing out the renders. The collective wisdom is that the write speeds are never a problem (it takes seconds to render one frame, but writing it out to disk is half a second). With this config, I can dump all my drives into one tower case, and have room for more if needed.

I'm a nuke newbie and am using the PLE version right now. I know after effects (I used it for keying, rotoing, motion graphics, grading and finishing my feature film 'The Impossible Murder' to an uncompressed HD TIFF sequence). I don't plan on doing any comping myself. I'll have to hire somebody who knows better. As for now, I've gone through all the free tutorials and videos on the foundry, the free fxphd series, etc. I might practise my skills so I can help my comper while he/she takes a break (I like doing this, no real necessity - I can just hire somebody else). I'm a director by profession but can't keep my hands off a computer. To answer your question, yes, I am planning to take the fxphd course in Nuke (simply because I can't find anything better).

Here's an update from Greg (DreamColor Solutions Architect) at HP:

The nVidia control panel allows to keep the display gamma at 1.0, and this should give us an uncorrupted output. I think I'll go with the Quadro 4000 after all.

Clearly, you have more knowledge in this area than me; so, could you please take a few minutes to explain why Quadro and gamma 1.0 is the desired setup for Nuke and Log & Linear compositing?

LOL. This is what I have learnt so far:

1. KNOW how the input files were created. Nuke guesses right most of the time, but most software don't work in linear space. In my case, I know I'll be getting
a. 10-bit DPX log files from the scanner (Cineon),
b. EXRs, FBX/OBJs (Linear), and
c. TIFFs in 32-bit (float) (sRGB).

Inside Nuke, everything is in linear space 32-bit RGB (Won't be using the log2lin node since I can control my read files).

2. The Quadro FX 4000 supports 10-bit RGB via DisplayPort. I will be using two monitors - both getting the same signal from the GPU.
a. The first one will be using the GUI and will have the viewer in sRGB mode. This, according to collective wisdom (ratified by Wikipedia!), is the worst color space to work in. But since I'm working with computers, I will use this as one reference. I will calibrate this according to sRGB specs independently. Real reason: It allows me to use a cheap 27" LCD monitor.
b. The second one will be the HP Dreamcolor, getting a 2K RGB 10-bit signal from the GPU. I will not be altering the GPU Gamma (leave it at one) or other settings that might affect the contrast, etc. I will use the in-built LUTs in the HP (after calibration) and also have a second profile (that has come from the DI facility). All I want to know is if the comps hold under different color spaces, including Rec 709 and PAL. Color accuracy is not paramount since I'm not using the best tools available to me. I understand that and accept the risks. Also in my case, this risk is warranted (see point 4) to a certain extent.

3. The important thing is the complete elimination of the HD-SDI pipeline, and any YUV signals that come with it. This is the only reason to invest in I/O cards like the Aja. The Dreamcolor engine shuts down when it gets interlaced and/or YUV signals.

4. I will be using Nuke for only two major sequences, and both of them are in 'their own worlds'. They don't have to match the rest of the film when it comes to color (except maybe skin tones). Everything else can be done in After Effects.

Unfortunately most of my knowledge is theoretical, but at least I've accomplished the objective of this thread, which was to get the hardware right. Now when the monitors arrive I will conduct as many tests as I can to avoid color issues. The single greatest lesson I've learnt is this: The lesser the number of monitors used for viewing, the more people agree on its output!

Isn't it funny how you come here looking for answers and yet, you end up teaching others?

With the dreamcolor, you should certainly be safe as far as colors go. I wonder if the DI facility has an ICC profile on hand for the 27" Dell. The main issue with the Dell is oversaturated Red, which proper calibration and profiling can eliminate. But, you really won't need the Dell for accurate color.

Just a heads up about DisplayPort: don't be surprised if the LCDs say there isn't any connection during boot-up. For adjusting anything during post (login to Raid or BIOS), connect a DVI to the Dell and just switch connections in the Dell. I have the same issue and thought my $6000 HP Z800 was broke when I got it and connected it to my Eizo for the first time. Not a fun feeling. Thus, I use my Dell connected via DVI for boot-up tasks.

DI facilities in India are scary places, something like Jurassic Park. I KNOW they won't have a clue as to what I'm aiming for. That's one more fight along the way.

Thanks for the heads up on the DisplayPort. I chose this over dual-link DVI because of the absence of screws, and the royalty-free indie spirit!

Your Z800 is sweet. It was my initial choice, but then I learnt the customer service for HP in India (although much better than other manufacturers) isn't reliable for critical issues. The systems you have are very close to what I want, the only big difference being the OCZ X2.

By the way, Kronos in NukeX seems like a great app, but in your experience, is there something better that I can get for cheaper? My particular use for it would be to add frames to slow down the footage (it uses the GPU). Nuke already has a node that does this (OFlow, FrameBlend). Since I don't have it in the PLE I can't compare.

Once again, thanks a lot for sticking with this thread and helping me out.

The Foundry sells Kronos for AE CS5 for only $100 plus there are rental options as well.

I compared a Mac Pro, Z800 and custom built options but I figured since Autodesk uses Z800s for their $50-200,000+ systems (Smoke-Flame), the Z800 should be rock-solid. Not to mention, I got 25% off WITHOUT even asking. Because you seem like a computer geek like me, here are its specs: dual 6-core Xeons @2.66GHz, Quadro FX3800, and purchased separately: 24GB ram ($1000), Intel X25 80GB SSD, 150GB Velociraptor for cache, 6 2TB Seagate Constellation ES with Areca 1680ix in Raid 5 for storage, 4 1TB Seagates in Raid 10 connected to onboard LSI SAS for rendered files (I do not recommend this LSI SAS due to very very low write speed).

My experience with HP support has been great so far thanks to a bad motherboard which did not regulate fan speeds causing the CPUs to constantly throttle during renders. I have only user their online tech support and each time, the tech seemed very knowledgeable.

PS Don't forget a good UPS, one that is true cine wave such as higher-end Cyberpower or APC. I have a nice APC at home and expensive Cyberpower at work for my Z800 and another for our broadcast server. At least in the US, 'true cine wave' is recommended for highly efficient power supplies. HP even states that some of their 85%+ efficient PSUs require this or the computer will shut off as demonstrated by numerous customer complaints.

Your machine is a beast. What kind of compositing work do you do with it - if you don't mind my asking?

Cooler Master Case + Power supply and APC UPS + Belkin Surge protector - won't forget those! I'll tell you a little secret - I was electrical engineer (once) - I know all about sine waves! I won't be making any compromises here.