transfering files at DVinfo.net

Ian Planchon · December 1st, 2008, 11:35 PM

I shot a 20 minute clip, plugged in the camera opened clip browser. it took 32 minutes to transfer the clip to the destination file.

is this because I am just plugging in the camera? would an express card slot on my computer be a good idea?

Paul Joy · December 2nd, 2008, 09:07 AM

It shouldn't take anywhere near that long. An expresscard slot would definitely be faster than USB but I see much faster speeds over USB than you're describing.

Ian Planchon · December 2nd, 2008, 09:52 AM

hmmm...my brand new computer still has some bugs, and took a crash the other day, there was a background raid initialization in progress, I bet that was slowing it down. I will try it again with the same files now that nothing is running and see....

Ian Planchon · December 2nd, 2008, 04:20 PM

so, this is lame. the camera is transfering files slower then real time when its a long continueous file...28 minutes took 36 to transfer....anyone else get that?

Dave Morrison · December 3rd, 2008, 12:10 AM

Hi Ian. I've been wanting to do some speed comparisons for a while now, and your bad luck with slow transfers spurred me to invest a couple hours tonight. I used an 8 gig card which had one continuous 28 min. clip on it. It was 7.1 GB and I transferred the entire BPAV folder to the desktop of both my desktop and laptop. I did my transfers via the following methods: (1) the Sony USB Card Reader, (2) the Express34 slot of my Macbook Pro (late '06 version) and (3) using the camera's USB transfer cable plugged into my Mac Pro desktop. That's it for the hardware.

For the actual transfers, I did: (1) a simple Finder "drag and drop", and (2) the Sony Clip Browser software (v2.00). In the Clip Browser transfers, I tried setting the Preferences two ways: the "safer" transfer method using "Perform CRC check after copy" turned on and "Data Protection" turned on, THEN the "faster" method with those preferences turned off ("Processing speed" checked and "CRC..." turned off). Here are my results:

LAPTOP TRANSFERS via Express34 slot
Finder Drag and Drop: 4:50
Clip Browser (fast): 5:47
Clip Browser (safe): 12:13

MACPRO via Sony USB Card Reader
Finder Drag and Drop: 7:04
Clip Browser (fast): 7:09
Clip Browser (safe): 15:58

CAMERA via USB transfer cable
Finder Drag and Drop: 10:35
Clip Browser (fast): 10:41
Clip Browser (safe): 19:17

As you can see, the ExpressCard slot is the fastest transfer of all. The Sony USB Card Reader isn't far behind while the camera's USB cable is the slowest.

dave

Andrew Stone · December 3rd, 2008, 02:20 AM

Interesting results Dave. I have been wondering why the Sony USB reader is not that far behind in speed to the Express Card ingest method. I think I've got it figured. Normally USB relies on the CPU to do a lot of the heavy lifting so there is a lot of back and forth between the USB device and the CPU with the CPU being handed a lot of processing calls. I am thinking the Sony USB device has onboard processors that do what normally would be handled by the CPU so there isn't the passing back and forth of processing thereby speeding up the ingest from the SxS card to the computer's hard drive. All conjecture but I cannot see how otherwise the Sony reader could have such a leg up on the Express Card coming relatively close to as fast in bringing the material into the computer. It just plain shouldn't be that fast.

Ian I really think you need local "hands on" help with your computer and it is beyond what a forum can offer. I had a quick look through your posts to figure out what system you are using and discovered you have had several problems getting your system up and running and may have done something to seriously compromise it's performance in the process of getting things configured and introducing additional hardware into the scenario.

Jeff Harper · December 3rd, 2008, 02:52 AM

Try foregoing raid for a minute, unistall your raid drivers and disable your raid controllers and see what happens.

Dave Morrison · December 3rd, 2008, 07:58 AM

I'm thinking about repeating all the tests but this time using only the Macbook Pro for ingesting. That would eliminate one variable that I didn't allow for, namely the bus speed differences between the desktop and the laptop. Not looking forward to repeating all the steps again, but I realize that my first effort might have introduced a variable that shouldn't be there.

Ian Planchon · December 3rd, 2008, 03:50 PM

Quote:

Originally Posted by Andy Stone

Ian I really think you need local "hands on" help with your computer and it is beyond what a forum can offer. I had a quick look through your posts to figure out what system you are using and discovered you have had several problems getting your system up and running and may have done something to seriously compromise it's performance in the process of getting things configured and introducing additional hardware into the scenario.

yeah, I am not a computer builder by any means. my bro in law did it. so I am thinking of taking it back to him and working out all the bugs. I really think its the raid, as when I sent the file to the desktop, it went in half the time.

Ted OMalley · December 3rd, 2008, 11:08 PM

Quote:

Originally Posted by Ian Planchon

yeah, I am not a computer builder by any means. my bro in law did it. so I am thinking of taking it back to him and working out all the bugs. I really think its the raid, as when I sent the file to the desktop, it went in half the time.

Ian,

That's why I mentioned what I did about your RAID 5 intentions last week. Going to a plain drive or a RAID 0 would be much faster. you usually don't want your video drives (the drives that temporarily store the large video files) to be working on redundancy as it really slows them down. Save the redundancy for the OS, for the project files (.veg files, in your case), and let the video drives run free and clear.

Andrew Stone · December 4th, 2008, 12:48 AM

Ian if you have a card which does RAID 5, I would suggest that you NOT abandon it. You will have paid more than a few hundred dollars for the card (I presume) and RAID 5 does not add that much of a performance penalty but if a drive does pack it in you will be able to rebuild the RAID in a few hours and move on. With a RAID 0 in that scenario you lose EVERYTHING and probably your life will be upside down as a result.

The peace of mind a fault protected RAID brings is huge. Don't underestimate it's importance.

I think your idea of rebuilding your RAID is a good starting point to troubleshoot but don't mess with your machine when the RAID is being built. You probably understand this now.

If you have nothing on your computer except installed apps I would suggest even reinstalling Windows and doing the machine all over again but really methodically and with someone who knows the ins and outs of windows and installing apps, drivers, utiilities and so on.

Jeff Harper · December 4th, 2008, 01:10 AM

I mentioned running with RAID controller disabled as a diagnostic tool. Did someone suggest Andy abandon RAID? I missed that.

I personnally had huge issues on my last two builds with raid controllers. But those controllers were the last thing I suspected.

The issues he's having seem to point to possible hard drive issues and by-passing the RAID is a logical place to begin.

Reinstalling windows before disconnecting your RAID would seem to be overkill when the symptoms appear to be what they are, which is hardware related. Of course it is possible re-installing windows may help. But by-passing the RAID takes all of 5 minutes and is much less work.

The simplest and safest way to do this is to remove the card, disconnect the RAID drives then reboot. When done with this phase of testing you simply reinstall the card and re-connect the drives and all should be as it was, there is no danger to the RAID configuration.

If operations improve after disconnecting your RAID controller you have just narrowed down your problem. If it doesn't improve you have eliminated your drives and controller as the culprit.

This is the first thing any qualified PC technician would likely do, but I could be wrong. I haven't used a PC technician since 1995, so I guess I don't know what they would do.

Larry Huntington · December 4th, 2008, 01:36 AM

Are you using a usb 2.0 cable? I noticed substantial loss in transfer times when using a usb 1.0 cable when transferring from the camera to the computer.

Ted OMalley · December 4th, 2008, 08:58 AM

Quote:

Originally Posted by Andy Stone

...RAID 5 does not add that much of a performance penalty...With a RAID 0 in that scenario you lose EVERYTHING and probably your life will be upside down as a result.

RAID's are great. RAID 1 is more reliable than a RAID 5 since in a RAID 5, if the controller fails, the content is not readable. In a RAID 1, the drive can always be moved to another machine to be read. Really, for performance and reliability, you want a RAID 1+0 (a.k.a. RAID 10) using four drives. This adds the redundancy of RAID 1 to the speed of RAID 0. When I ran a Raid 5 a couple of years ago, on two different video systems, I ran into nothing but problems - both performance problems which were more than minor drops, and failure problems causing a world of hurt.

RAID 5 was the ideal solution to the problem that used to exist where drives were not nearly large enough for the data. You could put ten drives in a RAID 5 and end up with a huge drives. Today, with 1TB drives nearing $100, that is much less of a problem. A RAID 1+0 providing 2TB of storage should be sufficient.

Also, Ian described to me that he was ONLY using these drives for the content that he'd load from the camera. He also said that he backed up the video to Blu-Ray discs. Meaning, if the project files are on the C:drive, and the RAID 5 is handling only the video that exists either on cards or on Blu-Ray, than the wrong data is fault-tolerant. Make the C: drive fault tolerant, and make the video portion fast. A loss of video data only takes copying it back from the Blu-Ray and you haven't lost a beat.

I still heartily don't recommend a RAID 5 - I'm not saying that others aren't having success, but in the RAID world I much prefer the selective use of RAID 0, RAID 1, and RAID 0+1.

Lastly, he doesn't necessarily have an expensive RAID 5 card - Many of the newer motherboards accept multiple drives and allow RAID 5 configuration - I own a few. This MAY be part of his problem.

Ian Planchon · December 4th, 2008, 09:33 AM

Quote:

Originally Posted by Ted OMalley

RAID's are great. RAID 1 is more reliable than a RAID 5 since in a RAID 5, if the controller fails, the content is not readable. In a RAID 1, the drive can always be moved to another machine to be read. Really, for performance and reliability, you want a RAID 1+0 (a.k.a. RAID 10) using four drives. This adds the redundancy of RAID 1 to the speed of RAID 0. When I ran a Raid 5 a couple of years ago, on two different video systems, I ran into nothing but problems - both performance problems which were more than minor drops, and failure problems causing a world of hurt.

RAID 5 was the ideal solution to the problem that used to exist where drives were not nearly large enough for the data. You could put ten drives in a RAID 5 and end up with a huge drives. Today, with 1TB drives nearing $100, that is much less of a problem. A RAID 1+0 providing 2TB of storage should be sufficient.

Also, Ian described to me that he was ONLY using these drives for the content that he'd load from the camera. He also said that he backed up the video to Blu-Ray discs. Meaning, if the project files are on the C:drive, and the RAID 5 is handling only the video that exists either on cards or on Blu-Ray, than the wrong data is fault-tolerant. Make the C: drive fault tolerant, and make the video portion fast. A loss of video data only takes copying it back from the Blu-Ray and you haven't lost a beat.

I still heartily don't recommend a RAID 5 - I'm not saying that others aren't having success, but in the RAID world I much prefer the selective use of RAID 0, RAID 1, and RAID 0+1.

Lastly, he doesn't necessarily have an expensive RAID 5 card - Many of the newer motherboards accept multiple drives and allow RAID 5 configuration - I own a few. This MAY be part of his problem.

you are correct. I dont own a raid card. I have a mobo that allows for raid 5. and as far as the backup, that was close, but not exactly what I am doing. the plan for HD (external) backup, and the older items get archived to bluray as they become more out of date. but yes, either way they are still backed up away from the raid.

December 1st, 2008, 11:35 PM	#1
Ian Planchon Major Player Join Date: Nov 2007 Location: an Alaskan living in Des Moines, Iowa Posts: 513	transfering files I shot a 20 minute clip, plugged in the camera opened clip browser. it took 32 minutes to transfer the clip to the destination file. is this because I am just plugging in the camera? would an express card slot on my computer be a good idea?

December 2nd, 2008, 09:07 AM	#2
Paul Joy Major Player Join Date: Mar 2007 Location: Norfolk, UK Posts: 627	It shouldn't take anywhere near that long. An expresscard slot would definitely be faster than USB but I see much faster speeds over USB than you're describing. __________________ Blog: www.pauljoy.com Company: www.videotrader.co.uk

December 3rd, 2008, 12:10 AM	#5
Dave Morrison Major Player Join Date: May 2002 Location: West Central Florida Posts: 762	Hi Ian. I've been wanting to do some speed comparisons for a while now, and your bad luck with slow transfers spurred me to invest a couple hours tonight. I used an 8 gig card which had one continuous 28 min. clip on it. It was 7.1 GB and I transferred the entire BPAV folder to the desktop of both my desktop and laptop. I did my transfers via the following methods: (1) the Sony USB Card Reader, (2) the Express34 slot of my Macbook Pro (late '06 version) and (3) using the camera's USB transfer cable plugged into my Mac Pro desktop. That's it for the hardware. For the actual transfers, I did: (1) a simple Finder "drag and drop", and (2) the Sony Clip Browser software (v2.00). In the Clip Browser transfers, I tried setting the Preferences two ways: the "safer" transfer method using "Perform CRC check after copy" turned on and "Data Protection" turned on, THEN the "faster" method with those preferences turned off ("Processing speed" checked and "CRC..." turned off). Here are my results: LAPTOP TRANSFERS via Express34 slot Finder Drag and Drop: 4:50 Clip Browser (fast): 5:47 Clip Browser (safe): 12:13 MACPRO via Sony USB Card Reader Finder Drag and Drop: 7:04 Clip Browser (fast): 7:09 Clip Browser (safe): 15:58 CAMERA via USB transfer cable Finder Drag and Drop: 10:35 Clip Browser (fast): 10:41 Clip Browser (safe): 19:17 As you can see, the ExpressCard slot is the fastest transfer of all. The Sony USB Card Reader isn't far behind while the camera's USB cable is the slowest. dave __________________ EX1, Nikon D750 http://www.morrisonphotographics.com http://www.heritagefilm.com

December 3rd, 2008, 07:58 AM	#8
Dave Morrison Major Player Join Date: May 2002 Location: West Central Florida Posts: 762	I'm thinking about repeating all the tests but this time using only the Macbook Pro for ingesting. That would eliminate one variable that I didn't allow for, namely the bus speed differences between the desktop and the laptop. Not looking forward to repeating all the steps again, but I realize that my first effort might have introduced a variable that shouldn't be there. __________________ EX1, Nikon D750 http://www.morrisonphotographics.com http://www.heritagefilm.com Last edited by Dave Morrison; December 3rd, 2008 at 08:39 AM.

December 2nd, 2008, 09:52 AM	#3
Ian Planchon Major Player Join Date: Nov 2007 Location: an Alaskan living in Des Moines, Iowa Posts: 513	hmmm...my brand new computer still has some bugs, and took a crash the other day, there was a background raid initialization in progress, I bet that was slowing it down. I will try it again with the same files now that nothing is running and see....

December 2nd, 2008, 04:20 PM	#4
Ian Planchon Major Player Join Date: Nov 2007 Location: an Alaskan living in Des Moines, Iowa Posts: 513	so, this is lame. the camera is transfering files slower then real time when its a long continueous file...28 minutes took 36 to transfer....anyone else get that?

December 3rd, 2008, 02:20 AM	#6
Andrew Stone Major Player Join Date: Sep 2008 Location: Vancouver, Canada Posts: 975	Interesting results Dave. I have been wondering why the Sony USB reader is not that far behind in speed to the Express Card ingest method. I think I've got it figured. Normally USB relies on the CPU to do a lot of the heavy lifting so there is a lot of back and forth between the USB device and the CPU with the CPU being handed a lot of processing calls. I am thinking the Sony USB device has onboard processors that do what normally would be handled by the CPU so there isn't the passing back and forth of processing thereby speeding up the ingest from the SxS card to the computer's hard drive. All conjecture but I cannot see how otherwise the Sony reader could have such a leg up on the Express Card coming relatively close to as fast in bringing the material into the computer. It just plain shouldn't be that fast. Ian I really think you need local "hands on" help with your computer and it is beyond what a forum can offer. I had a quick look through your posts to figure out what system you are using and discovered you have had several problems getting your system up and running and may have done something to seriously compromise it's performance in the process of getting things configured and introducing additional hardware into the scenario.

December 3rd, 2008, 02:52 AM	#7
Jeff Harper Inner Circle Join Date: Jun 2005 Location: Cincinnati, OH Posts: 8,425	Try foregoing raid for a minute, unistall your raid drivers and disable your raid controllers and see what happens.

December 4th, 2008, 12:48 AM	#11
Andrew Stone Major Player Join Date: Sep 2008 Location: Vancouver, Canada Posts: 975	Ian if you have a card which does RAID 5, I would suggest that you NOT abandon it. You will have paid more than a few hundred dollars for the card (I presume) and RAID 5 does not add that much of a performance penalty but if a drive does pack it in you will be able to rebuild the RAID in a few hours and move on. With a RAID 0 in that scenario you lose EVERYTHING and probably your life will be upside down as a result. The peace of mind a fault protected RAID brings is huge. Don't underestimate it's importance. I think your idea of rebuilding your RAID is a good starting point to troubleshoot but don't mess with your machine when the RAID is being built. You probably understand this now. If you have nothing on your computer except installed apps I would suggest even reinstalling Windows and doing the machine all over again but really methodically and with someone who knows the ins and outs of windows and installing apps, drivers, utiilities and so on.

December 4th, 2008, 01:10 AM	#12
Jeff Harper Inner Circle Join Date: Jun 2005 Location: Cincinnati, OH Posts: 8,425	I mentioned running with RAID controller disabled as a diagnostic tool. Did someone suggest Andy abandon RAID? I missed that. I personnally had huge issues on my last two builds with raid controllers. But those controllers were the last thing I suspected. The issues he's having seem to point to possible hard drive issues and by-passing the RAID is a logical place to begin. Reinstalling windows before disconnecting your RAID would seem to be overkill when the symptoms appear to be what they are, which is hardware related. Of course it is possible re-installing windows may help. But by-passing the RAID takes all of 5 minutes and is much less work. The simplest and safest way to do this is to remove the card, disconnect the RAID drives then reboot. When done with this phase of testing you simply reinstall the card and re-connect the drives and all should be as it was, there is no danger to the RAID configuration. If operations improve after disconnecting your RAID controller you have just narrowed down your problem. If it doesn't improve you have eliminated your drives and controller as the culprit. This is the first thing any qualified PC technician would likely do, but I could be wrong. I haven't used a PC technician since 1995, so I guess I don't know what they would do.

December 4th, 2008, 01:36 AM	#13
Larry Huntington Major Player Join Date: Nov 2005 Location: Grass Valley, California Posts: 350	Are you using a usb 2.0 cable? I noticed substantial loss in transfer times when using a usb 1.0 cable when transferring from the camera to the computer.