HDlink and CF output use only one core? at DVinfo.net
DV Info Net

Go Back   DV Info Net > Cross-Platform Post Production Solutions > CineForm Software Showcase

CineForm Software Showcase
Cross platform digital intermediates for independent filmmakers.

Reply
 
Thread Tools Search this Thread
Old March 17th, 2011, 03:52 PM   #1
Major Player
 
Join Date: Oct 2003
Location: Portland OR
Posts: 227
HDlink and CF output use only one core?

Clearly CS5 + Neo4K and and a suitable Nvidia card are a huge improvement over CS4. I do note however that both HDlink conversions of .mts to .avi and AME Cineform renderouts use only 25% of my quad. The performance monitor show HDlink running 35 threads....but at 22-25% it looks like it has not engaged my other 3 cores. Similar utilization with CF output from PProCS5 when all video adjustments done in FirstLight. All else seems to provide no bottlenecks....is there some setup item I missed or is the app just not multi-threaded enough?

Details
ASUS P6TDeluxV2 mobo, x58 chipset and 3banks x 2 strips for RAM
running 12GB Kingston 1333 DDR3 as 6 strips
i7-875 quad at 3.33ghz and the Win7 x64 device manager shows 8 processors
very effective cooler and case venting
OS on single 7200rpm SATA disc
Video and all work files on striped pair 7200rpm SATAs
and show only abt 15MBps throughput (copies can go way over 50MBps)
Nvidia FX 3800 which CS5 does recognize

[Keep up all those great updates! - Don]

Last edited by Don Blish; March 17th, 2011 at 08:16 PM.
Don Blish is offline   Reply With Quote
Old March 17th, 2011, 09:58 PM   #2
CTO, CineForm Inc.
 
Join Date: Jul 2003
Location: Cardiff-by-the-Sea, California
Posts: 8,095
Re: HDlink and CF output use only one core?

No, we are fully threaded. HDLink should regularly saturate all you cores if your disks can keep up (I can saturate 12 cores.) CS5 export is just slow for CineForm exports, our encoding is twiddling its thumbs waiting for frames form Premiere.
__________________
David Newman -- web: www.gopro.com
blog: cineform.blogspot.com -- twitter: twitter.com/David_Newman
David Newman is offline   Reply With Quote
Old March 19th, 2011, 03:20 PM   #3
Major Player
 
Join Date: Feb 2008
Location: Voorheesville, NY
Posts: 433
Re: HDlink and CF output use only one core?

Quote:
Originally Posted by Don Blish View Post
... Video and all work files on striped pair 7200rpm SATAs
and show only abt 15MBps throughput (copies can go way over 50MBps)
How are you measuring real time disk throughput while encoding? I have a dual CPU (12 physical, X5650) Xeon workstation and when I read and write from independent striped pairs of 7200 rpm SATA drives, I get the same CPU load (somewhere around 15-18%), so I suspect that what David said is correct, the disk subsystem isn't keeping up with the CPUs. Benchmarking my drives with ATTO Disk Benchmark, I get ~ 150 MB/s reads and writes and I know these numbers are accurate, since I can capture full HD with my Blackmagic Design capture card without dropping frames.

Just as an aside, to some extent how many cores that a video encoder uses also depends on the decoding. HDLink will use a lot more of the CPU power, if it encodes either an MPEG2 or h264 file to CFHD.
Jay Bloomfield is offline   Reply With Quote
Old March 20th, 2011, 04:34 PM   #4
Major Player
 
Join Date: Oct 2003
Location: Portland OR
Posts: 227
Limited Disc Thruput?

For disc thruput measures I just use Win7-x64s Task Manager and Resource Monitor tab. My video striped pair are WD Caviar-Green 1.5TB, 7200rpm discs SATA connected to the Intel ICH10R ports on the MoBo. They show about 30MB/sec during either HD link (AVCHD to .avi) or CS5 Cineform Exports. Using that "overview" tab just now on a 6GB copy from the striped pair to my single 7200rpm root drive, the overview showed 90GB/sec. From the "Disc" tab you could see that it was 45GB/sec for each drive.

Perhaps I need two striped pairs, one for the source and one for the target of such efforts.
Don Blish is offline   Reply With Quote
Old March 20th, 2011, 07:28 PM   #5
Major Player
 
Join Date: Feb 2008
Location: Voorheesville, NY
Posts: 433
Re: HDlink and CF output use only one core?

The only other "bottleneck" that I can think of is that disk writes are much slower, if the actual size of data that is written in each write cycle is smaller. For example, my striped pairs will support 150 MB/s transfers, but only above 64 KB transfer sizes. You would think that the CFHD encoding chain would be writing large amounts of data at a time to the hard drive, though.

Also, I assume that you meant MB/s. BTW, I also see a maximum hard disk transfer rate of 50 MB/sec when encoding CFHD, which is well below the max that I see with other software, including OS copies from one drive to the other..
Jay Bloomfield is offline   Reply With Quote
Old March 21st, 2011, 07:42 PM   #6
Major Player
 
Join Date: Feb 2008
Location: Voorheesville, NY
Posts: 433
Re: HDlink and CF output use only one core?

Here's an update (Sorry, I couldn't edit the previous post). I created a RAM Disk (since I have 24 GB RAM) and tried to transcode an mt2 file to CFHD. Now, the RAM Disk has read and write transfer speeds that are at least an order of magnitude higher than even the fastest consumer SSD (I get 2.7 GB/s). Even with putting both the input and output files on the RAM Disk, the maximum CPU usage on my computer was 35%, indicating that the disk subsystem is not the cause of the bottleneck. I personally don't think that this result has anything to do with HDLink, but rather either has something to do with the OS or with the nature of video transcoding in general and how amenable it is to hyperthreading with large numbers of threads.

RAMDisk Enterprise is shareware, so if you want to test your system, you can download it from the author's website:

RAMDisk for Windows 2000 / XP / Server 2003 / Vista / PE
Jay Bloomfield is offline   Reply With Quote
Old March 21st, 2011, 07:59 PM   #7
CTO, CineForm Inc.
 
Join Date: Jul 2003
Location: Cardiff-by-the-Sea, California
Posts: 8,095
Re: HDlink and CF output use only one core?

No one has discussed what they are converting. If the source format uses a third decoder that is single threaded, that will be your bottleneck. Becareful installing tools like FFDShow as you source decode may change from the one we ship to something else on you system.
__________________
David Newman -- web: www.gopro.com
blog: cineform.blogspot.com -- twitter: twitter.com/David_Newman
David Newman is offline   Reply With Quote
Old March 21st, 2011, 09:06 PM   #8
Major Player
 
Join Date: Feb 2008
Location: Voorheesville, NY
Posts: 433
Re: HDlink and CF output use only one core?

I get a somewhat better result "transcoding" Cineform to Cineform using HDLink (just resizing), I get CPU usages in the range of 50-60%, using a RAM Disk for both input and output. In any case, whatever is causing this behavior (as reported by the OP, I was only checking on it, because I never thought much about it before) on dual Xeon Win 7 x64 systems does not inherently have anything to do with the disk subsystem. My result is with build 308 of NEO 5 4K.
Jay Bloomfield is offline   Reply With Quote
Old March 22nd, 2011, 12:32 PM   #9
Major Player
 
Join Date: Oct 2003
Location: Portland OR
Posts: 227
HDlink "convert"

Quote:
Originally Posted by David Newman View Post
No one has discussed what they are converting. If the source format uses a third decoder that is single threaded, that will be your bottleneck. Becareful installing tools like FFDShow as you source decode may change from the one we ship to something else on you system.
As quickly said in my post, I was converting an .mts file in AVCHD format from my Pany HDC-TM300. Could the .MTS "reader" be the bottleneck?

[The other "complaint" about CS5 cine renderout you explained was that your software was waiting on CS5 to offer up the frames].
Don Blish is offline   Reply With Quote
Old March 22nd, 2011, 12:38 PM   #10
CTO, CineForm Inc.
 
Join Date: Jul 2003
Location: Cardiff-by-the-Sea, California
Posts: 8,095
Re: HDlink and CF output use only one core?

My 6 core 12-HT Gulftown system converts Panasonic MTS files with 70% CPU usage. MTS is not the issue.
__________________
David Newman -- web: www.gopro.com
blog: cineform.blogspot.com -- twitter: twitter.com/David_Newman
David Newman is offline   Reply With Quote
Old March 22nd, 2011, 08:46 PM   #11
Major Player
 
Join Date: Feb 2008
Location: Voorheesville, NY
Posts: 433
Re: HDlink and CF output use only one core?

It's getting harder and harder to figure out what codecs are being used in what instances and the GraphEdit "replacement", Microsoft's Media Foundation Topology Editor doesn't work with all codecs. This "codec fog" is true, even for a system like mine that has very little "fluff" installed, except for programs like Vegas and CS5.

Maybe this is a bad example, but I created a graph for a Blackmagic Design AVI file with GraphEdit. Using the Microsoft MJPEG decoder, only one thread is used. If I edit the graph and use the Decklink MJPEG codec, it is obviously multithreaded (using a Win 7 gadget that displays all thread activity). I created both x86 & x64 graphs and the results were the same.

I wonder how heavily threaded the rest of the built-in Microsoft codecs are?
Jay Bloomfield is offline   Reply With Quote
Old March 23rd, 2011, 07:14 AM   #12
Major Player
 
Join Date: Dec 2005
Location: Natal, RN, Brasil
Posts: 900
Re: HDlink and CF output use only one core?

Quote:
Originally Posted by Jay Bloomfield View Post
...RAMDisk Enterprise is shareware, so if you want to test your system, you can download it from the author's website:

RAMDisk for Windows 2000 / XP / Server 2003 / Vista / PE
This URL is apparently no longer valid, Jay. Any suggestions on how to find this now?
__________________
http://lightinaction.org
"All in the view of the LION"
Stephen Armour is offline   Reply With Quote
Old March 23rd, 2011, 08:03 PM   #13
Major Player
 
Join Date: Feb 2008
Location: Voorheesville, NY
Posts: 433
Re: HDlink and CF output use only one core?

I just tried it. It works for me. Maybe the site was just down for a while. If you can't find it on the author's site, you can find it with Google, etc..

Aside from the issue raised in the OP of this thread, a RAM Disk is also helpful to speed up any operation where an intermediate file is used. You have to keep in mind that such storage is volatile (the contents are lost when you reboot) and even 24 GB of RAM isn't really that big these days, when looking at the size of some video files. RAM Disks can be expanded and contracted on the fly, although some times it requires a reboot.
Jay Bloomfield is offline   Reply With Quote
Old March 24th, 2011, 07:35 AM   #14
Major Player
 
Join Date: Dec 2005
Location: Natal, RN, Brasil
Posts: 900
Re: HDlink and CF output use only one core?

Weird. I still can't access it and couldn't find it Googling it! We're located in Northeastern Brazil, so who knows what's up.

We've been having some really strange DNS server probs for weeks and I've been swapping new DNS servers in and out of our router to try to re-route to troublesome URLs.

Attached is the screenshot I'm getting. If you've got a direct link to their download, I'd appreciate it.
Attached Thumbnails
HDlink and CF output use only one core?-screenshot.jpg  
__________________
http://lightinaction.org
"All in the view of the LION"
Stephen Armour is offline   Reply With Quote
Old March 24th, 2011, 03:30 PM   #15
Major Player
 
Join Date: Feb 2008
Location: Voorheesville, NY
Posts: 433
Re: HDlink and CF output use only one core?

@Stephen,

I PMed you with the links, as I didn't want this thread to go too far off topic. :-)
Jay Bloomfield is offline   Reply
Reply

DV Info Net refers all where-to-buy and where-to-rent questions exclusively to these trusted full line dealers and rental houses...

B&H Photo Video
(866) 521-7381
New York, NY USA

Scan Computers Int. Ltd.
+44 0871-472-4747
Bolton, Lancashire UK


DV Info Net also encourages you to support local businesses and buy from an authorized dealer in your neighborhood.
  You are here: DV Info Net > Cross-Platform Post Production Solutions > CineForm Software Showcase

Thread Tools Search this Thread
Search this Thread:

Advanced Search

 



All times are GMT -6. The time now is 02:44 PM.


DV Info Net -- Real Names, Real People, Real Info!
1998-2024 The Digital Video Information Network