can you recommend hd (RAID?) array? Transfer/editing workflow?

[Malcolm Hamilton]

Hi there,
I currently edit SD (on a Mac powerbook, using Avid Xpress Pro and an assortment of fw drives), but plan to get an EX1 and a MacBook Pro very soon, and want to start editing HD.
I've been told that you can edit HD with fw 800 drives, but a friend strongly advised RAIDED SATA drives - - this would mean eSATA for me I guess.
Would people on this forum be kind enough to advise me as to what my best purchasing decision/workflow would be re hard drives?
I've read lots of very good things about CalDigit's HD Pro set up... it's very expensive, but CalDigit just came out with a cheaper, extremely fast set-up called HDOne. It has removeable drives (up to 8 of them), built-in PCI Express, supports RAID 5 and 6...
It SOUNDS very good to me, but I'd love some advice.
If RAIDED SATA drives ARE the way to go... what would my best modus operandi be...
Would I come back from a shoot, transfer footage from my MacBook Pro to the eSATA drives AND AT THE SAME TIME transfer to a fw drive (as a backup)? Can you do this at the same time?
Thanks for any advice.
Malcolm

[Mark Edwards]

Well, aside from caldigit, there is
http://maxxdigital.com/shop/index.php?cPath=58_78
I have heard these drives are some of the best and they are slightly cheaper than most other brands. They may not look like much, but its the inside that counts, right.
http://www.dulcesystems.com/
This company also makes some great raid setups.
g-technology.com
these are great also and are more expensive.

All of these are great options that you can't go wrong with. Just make sure you check your system specs and know what your getting into. They all offer raid protection.

[Dean Harrington]

You might check this out: http://www.amazon.com/Micronet-MDN10.../dp/B000VEJGXC
I'm using 2/ 500 gig. Sata drive OWC kit. They are not raided but I've had no problems with this over the last 1 year or more with my MacBook Pro. The G Force was recommended by a good friend and editor, so, I thought I'd pass it on. I don't have it so can't comment on it's stability.

[Evan Donn]

Originally Posted by Malcolm Hamilton

I've been told that you can edit HD with fw 800 drives, but a friend strongly advised RAIDED SATA drives - - this would mean eSATA for me I guess.

Are you planning to edit XDCAM native, or convert to another format? I'm not convinced there's a significant advantage to an eSATA RAID with XDCAM, especially if you're editing on a Macbook Pro. 35mb/s is less than 5Mb/second - the latest 1GB SATA drives are sustaining 75-100Mb/s read/write speeds (http://www.barefeats.com/hard94.html). Thus a single one of those drives in a fw800 enclosure should easily support 10+ streams of XDCAM, which is more than the processor in the Macbook can decode smoothly.

If you're converting to ProRes it makes more sense, but even then it's not absolutely necessary. At the lower data rate, 145mb/s or ~18Mb/s you should still be able to comfortably run 3-4 streams from a single drive, and at the higher data rate (220mb/s or ~27Mb/s) at least 2 streams.

[Malcolm Hamilton]

Originally Posted by Evan Donn

Are you planning to edit XDCAM native, or convert to another format? I'm not convinced there's a significant advantage to an eSATA RAID with XDCAM, especially if you're editing on a Macbook Pro

Hi Evan, and Mark and Dean - -

Thanks so much for getting back to me. I understand what you're saying, Mark and Dean... it seems there are a number of good drive-makers out there (I know G-Tech is good, but I did have a G-Tech fw drive fail on me a few months ago... which is why I'm now smitten with the idea of RAID).

Evan, I get a bit lost when it comes to your advice (my fault, not yours)... but I think I get the drift: eSATA might be more than I need (i.e., faster than what the MacBook Pro is capable of), but here's the thing - - I'd like to ”future-proof“ myself a little bit... e.g., if I get a Mac Pro six months from now (honestly, it's the Mac Pro I WANT to get for editing... I'm getting the MacBook Pro just for the sake of being able to download in the field; if there were an inexpensive way to download in the field, I'd use it, and do my editing on a Mac Pro back in my humble office), I don't want to have to upgrade my hard drive “array”.

I want to come up with a solid solution not just for now... but (if this is even possible) for the next few years.

I get lost re the number of ”streams“ of data I'll be running (oh, and re editing XDCAM native... does this mean the 35Mb/s stuff? If so, yes, I'd like to be able to, if I can... can I?)... but aside from that... given that I might be adding a Mac Pro to my equipment list in six or eight months time... does eSATA make sense? I don't know if it makes a difference (re your reference to ProRes) that I use Avid Xpress Pro as my main nle...
Thanks so much for your help on this,
Malcolm

[Dean Harrington]

One reason I mentioned G Force raid is that it's ethernet cable. As far as I know, someone correct me if I'm wrong, it's the fastest of all the connections.
http://www.answers.com/topic/g-force-raid

[Noah Yuan-Vogel]

Originally Posted by Evan Donn

Are you planning to edit XDCAM native, or convert to another format? I'm not convinced there's a significant advantage to an eSATA RAID with XDCAM, especially if you're editing on a Macbook Pro. 35mb/s is less than 5Mb/second - the latest 1GB SATA drives are sustaining 75-100Mb/s read/write speeds (http://www.barefeats.com/hard94.html). Thus a single one of those drives in a fw800 enclosure should easily support 10+ streams of XDCAM

Are you sure about the streams? my understanding of hard drives, especially single ones, is that those sustained speeds are for sequential data only. that means if you are dealing with more streams at one time your drive will be seeking, bringing your 100MBps Hard drive down as low as 3MBps.

[Noah Yuan-Vogel]

Originally Posted by Dean Harrington

One reason I mentioned G Force raid is that it's ethernet cable. As far as I know, someone correct me if I'm wrong, it's the fastest of all the connections.
http://www.answers.com/topic/g-force-raid

what do you mean? gigabit ethernet? its about 2-4x slower than esata, and in most consumer NAS'es it is, in reality, about 10x slower.

personally id say, if you need a lot of speed for cheap, esata is absolutely the way to go. and if you can afford a RAID, it is absolutely worth it if you are doing a lot of editing. the extra speed, capacity, and security they provide (depending on the configuration) are extremely useful. the only problem is finding exactly the right one for your needs (extra redundancy, extra speed, portability, ease of use, low price, etc)

[Evan Donn]

Originally Posted by Noah Yuan-Vogel

Are you sure about the streams? my understanding of hard drives, especially single ones, is that those sustained speeds are for sequential data only. that means if you are dealing with more streams at one time your drive will be seeking, bringing your 100MBps Hard drive down as low as 3MBps.

On the link I provided ( http://www.barefeats.com/hard94.html ) you can see the drop in speed for small random vs. large sequential - it is significant, cutting the read speeds into the 30MBs range - which is still plenty for multiple streams of XDCAM. My experience with HDV has been that for any single dual-core processor (i.e when editing on a Macbook) the bottleneck is the processor's ability to decode multiple streams rather than the drive's ability to stream them.

On an 8-core mac pro the situation would be reversed and there would be an advantage to an eSATA array if you are doing a lot of compositing with multiple streams of video. If you're mostly doing basic editing I don't know that you'll see a significant difference.

Originally Posted by Malcolm Hamilton

I'd like to ”future-proof“ myself a little bit... e.g., if I get a Mac Pro six months from now (honestly, it's the Mac Pro I WANT to get for editing... I'm getting the MacBook Pro just for the sake of being able to download in the field; if there were an inexpensive way to download in the field, I'd use it, and do my editing on a Mac Pro back in my humble office), I don't want to have to upgrade my hard drive “array”.

I want to come up with a solid solution not just for now... but (if this is even possible) for the next few years.

Absolutely, that makes sense, and as I mentioned above you would potentially see an advantage with a Mac Pro and eSATA. However, whenever it comes to future-proofing my advice is to be realistic about your time frame. Are you going to get a Mac Pro in six months? If not - if it's closer to a year or more - you should probably just worry about what you need right now, not what you'll need then. In a year you'll be able to get a faster array with greater capacity for less money.

oh, and re editing XDCAM native... does this mean the 35Mb/s stuff? If so, yes, I'd like to be able to, if I can... can I?)... but aside from that... given that I might be adding a Mac Pro to my equipment list in six or eight months time... does eSATA make sense? I don't know if it makes a difference (re your reference to ProRes) that I use Avid Xpress Pro as my main nle...

Yes, XDCAM native is 35 megabits/second or approximately 4.5 megabytes/second. In terms of drive performance that's probably the easiest format to edit in. I don't really know Avid - my understanding is DNxHD is comparable to ProRes in terms of data rate, but I don't know if Xpress Pro supports it.

[Simon Cox]

I use the Sonnet Fusion F2 eSATA Raid drive. Its a 640GB RAID drive in a tiny enclosure (2 X 320GB drives in there) and takes power from your Firewire port on the laptop. It connects to the Macbook Pro via an expresscard 34 eSATA card - so my workflow is thus: shoot, then insert the SxS card into the Macbook Pro directly - copy the card to a portable LaCie rugged drive via Firewire 800 - this acts as my backup - then swap the SxS card for the eSATA card, connect the Fusion F2 - copy across the files from the rugged drive to the Fusion and start editing.

Its a great solution for me - means that I have an instant backup and I can edit in the field without the need for a separate power source.

Simon Cox

[Dean Harrington]

Originally Posted by Noah Yuan-Vogel

what do you mean? gigabit ethernet? its about 2-4x slower than esata, and in most consumer NAS'es it is, in reality, about 10x slower.

personally id say, if you need a lot of speed for cheap, esata is absolutely the way to go. and if you can afford a RAID, it is absolutely worth it if you are doing a lot of editing. the extra speed, capacity, and security they provide (depending on the configuration) are extremely useful. the only problem is finding exactly the right one for your needs (extra redundancy, extra speed, portability, ease of use, low price, etc)

Looks like eSata will transfer 6 gigs per second and ethernet 1 gig per second according to the literature! Of course, I wonder what people are really getting? I'm using a Mac Book Pro intel with a teribit eSata drives in a kit set-up and they are fast but ... don't know the exact speeds.

[Malcolm Hamilton]

Evan - thank you for your thoughtful advice.
I understand now that, with a MacBook Pro, the eSATA speeds of a RAID array (like the HDOne) might be wasted... any fw (800 anyway) drive would be just as fast.
(this, I presume, would apply not just to basic editing, but also rendering effects and this sort of thing?)

But what about security (eg., RAID 5)? I had a firewire drive fail on me a few months ago (I lost three weeks of work), and swore that my next hd setup would be RAIDed so that this would never happen again.
If fw 800 is just as fast as RAID, what do people do for peace of mind... just make a habit of always transferring footage to two fw drives?

If the above is a perfectly good working solution, I might just get a few more fw 800 drives (I guess I could use my collection of fw400 drives as my ”backup/redundancy“ drives)... but if I DO end up going eSATA because of my plans to get a Mac Pro in the fall (or by December, anyway), I have two questions for Simon, or someone like Simon who uses eSATA... does swapping an SXS card for an eSATA card involve a re-boot?
And... do they make ExpressCard hubs... so you don't have to swap?

Thanks so much,
Malcolm

[Simon Cox]

All hot swappable so no need to reboot between the SxS card and the Fusion F2 for me.

[George Kroonder]

... but do unmount the volumes first; don't just yank the card!

George/

P.S. If you find any ExpressCard extenders (hubs), I'd like to know. I haven't turned up any.

[Mark Huang]

Malcolm:

I have a 6 TB Caldigit HDPro for a few weeks now. I got my from B&H New York. It works great so far. HDOne is similar to HDPro but with with some cost saving measures. (1) HDOne does not have a removable RAID card. (2) The memory on the RAID card is not ungradlbe. (3) No redundant power supply (I think).

Caldigit is retiring its S2VR HD.

Mark