Raid-0 Expectations
 

I'm finally going RAID-0 on my Quad w/ 6gb memory. I'm installing two VelociRaptor 300 internals @ 10000rpm. Until now I've relied on firewired externals.

Question: With this setup will I be able to simultaniously render HDV to the VelociRaptors while capturing HDV with a networked Macbook pro (also to the VelociRaptors)?

I don't want any gremlins, but capture time is killing me in producing three-camera weddings.

Dana,

Unless there's an adapter I'm not aware of you can't install the VelociRaptor in a MacPro or any other RAID-tower/server setup with removable trays; the SATA/power connectors have moved from the normal lower-side position to squarely in the middle of the drive - they wil not match up with where those connectors on the MacPro mainboard. Sorry.

Also, you're not going to get good results by creating an internal RAID; if you're serious about increasing drive performance you're going to get a *real* external RAID enclosure, whether it's eSATA, SCSI or Fiber.

Thanks Robert,

There is an adapter, and Western Digital told me it had to be internal, but I believe you over them. What would you recommend? We've got to be able to capture and edit at the same time, and to the same drive(s). I was impressed by the 10000 rpms of the Velociraptor...

WD is correct; the only way the V-raptor can connect is with a standard SATA/power cable going to an internal SATA controller card or to an addtional SATA mainboard connector. However in a MacPro this won't work because there's no way to put any addtional connectors on the back of the drive because the drive only has room to interface with the built-in connections on the "backplane" of the MB. You simply can't use the V-raptor in a MacPro, it would have to be in an external enclosure connected via eSATA host card.

You *can't* capture AND edit to the same drive at the same time; the only way you could pull that off is if you had a very large and *fast* RAID array with a LOT of bandwidth to handle two systems hitting the same array at the same time. (I'm assuming you're talking about using more than one Mac, right?)

That means you'd need either a SCSI or Fiber array because no eSATA array of any size would have the ability to handle all the HDD cache required for multi-user editing.

My suggestion would be to get with one of the A/V suppliers and or a FCP specialist locally and contract them to help you figure out how to accomplish what you need. Without knowing your budget and your exact workflow - and exactly what you're trying to accomplish it would be difficult to make direct suggestions.

to capture and edit at the same time, you just need to capture to another disk, whatever speed it is, no needs for RAID.
it is not even sure that a fast RAID would be ok to capture and edit on same disks, because you can be never sure if the sustained write would not drop below requirement if a big sustained read would occur at same time.
with two disk , there would be no question.

Robert, can you elaborate on this some??? Is this the differnce between a internal software RAID vs an external hardware RAID??? If not, what is the limitation for an internal RAID??? Also, what would you expect to be a reasonable data rate an internal RAID be capable of supporting??? I know, I know... too many variations... but could you ballpark it??? And how would this compare to an external RAID performance???

Mark

(laughs) Yes, many questions but I'll try to simplify:

The best you can hope for (real world numbers, not the hyped-up theoretical numbers) in an internal array is around 110-130 MB/s, but that's not a constant because just like with external eSATA arrays there is no controller cache available to offload the HDD cache when they fill up, so as you work on edits the data-rate will slow down to about the speed of a single HDD.

There's tons of info (including tests that I've done) posted on the forum; if you use the search feature you'll find more info than you handle.

The only real benefit to creating in internal RAID is to make one large single-partition, but it will not be fast. If you do go that route DO NOT put your OS on an internal array, make sure your boot-drive has it's own physically separate HDD, not a partition on the array.

A raid is a raid. Does not matter if it is internal or external.
The bottleneck is at different place. Depends your hardware.
the first one could be the disk. An average SATA 300 drive is about 60Mb/sec so you can expect with Raid 0 a minimum of 120 Mb/sec.
For example my Shuttle barbone X38Pro uses 2 Western digital 320 gig sata drive in Raid 0
they are average quality disks and the Blackmagic utility for testing disk speed reads 145Mb/s.
that is on an empty disk. a full disk can loose as much as 50% of its speed. so you better had to plan twice the capacity needed so you are sure to never end up with full/slow disks.
in my case , a got 2x320=over 600gig of disk space, so i am ok in the first 200gig.
(i am supposed to capture components directly out from the camera, bypassing HDV compression).
when i reach this capacity, the captured clips are transferred to another drive (one internal and one external for backup).
This pc has also E-Sata plug (one the same controller, so there would be no difference if the disk would be inside or outside the case).
The second bottleneck could be the controller . if the card is on a PCI bus (limited to 133Mhz) there no chance you could get faster than a SATA150 disk could give.
if the chip is on the motherboard, you have to check how it is linked the the CPU (kind of chipset used)
It can use a PCI bus, a PCIe bus , at 1x or 16X. It could share the bus (bandwith) with something else (USB controller) .
here is a picture of the architecture of my chipset (X38)
http://www.intel.com/Assets/Image/di...ck_Diagram.jpg
here you can see that if any link between a SATA connector is 3G/s, the main link between the other part of the chipset is limited (and shared with USB, audio, Ethernet ) to 2G/s .
that is what is called Northbridge and Southbridge by chipset specialists.



The last bottleneck, is the way the RAID is managed, the kind of command it can handle, some controller being a way better than other just because the firmware.

but again, even the fastest raid could get problem if the head must read video and write at the same time. the displacement of the head will probably be so heavy, that you will not be able to get sustained bandwidth either for read or for write.
While a simple cheap harddisk dedicated for capture will do the trick , even on an USB2 port.

Hmmm... I've got a 5 disk internal RAID 0 using bays 2 thru 4 and the 2 extra SATA ports with drives located in the spare optical drive bay. The drives are WD2500YS enterprise drives forming a 1.1TB RAID. It's about 2/3 full. Using the BM Disk Speed Test I'm getting about 290MB/s reads and writes. In real terms, I've been able to capture 10 bit 4:2:2 YUV 30fps uncompressed without dropping a frame. But I'm always looking for more, so your comments made me wonder.

There are a few third-party kits that allow you to install up to (8) SATA drives in a MacPro however this is not an ideal situation. (Be very wary of the "MaxUpgrades" kits as they are *not* approved by Apple and there have been documented MB failures as a result of those "upgrades")

That scenario is not "hot-swap", so if you have a drive failure you've got a lot of work to do to replace it. You're also at a near-overload situation with respect to the internal power supply for the machine not to mention adding unplanned-for extra heat to the entire tower that has to be moved out. HDD's generate a great deal more heat than optical drives and the airflow routing for that area was not designed to handle that additional heat.

Mac's have always been very carefully designed and Apple has several white-papers about the data-flow design of the MB, heat-dissipation, and power-supply load-levels.

With respect to your data-rates, they will initially test very high but as your drives are used during an edit the data-rate will drop because there is no hardware RAID controller offering mid-level cache to offload the internal HDD cache.

Just because an accessory has been created for internal modification doesn't mean it's well thought out. In fact, Apple has clearly stated that certain internal upgrade "kits" will void both the standard and APP warranties if installed. Buyer beware.

again , having one Ferrari is useless if you need to go at to different location at the same time. Getting 2 cheap cars will be more efficient. (or getting 2 ferraris even better).
in your case, getting 2 raid (one for capture and one for editing ) will be more efficient that having only one superfast raid.
I never understood the Blackmagic utility screen. do they display in Pic/sec or frame /sec ?
for example if it says 37 in the 1080, doest it means you are ok (over 30 im/sec) or do you need to have over 60 (frame/sec) ?

I don't see the need for a Raid-0 for capturing. It's realtime, so my firewired extrernals do fine. Rendering is where Raid-0 would shine, but I'm not sure what to look for. It sounds like I need to find an external Raid-0 enclosure w/a firewire 800 out. (recommendations?)

If I capture to a HDD (A) and render to the Raid-0 extrenal (B), can I capture again to (A) while (B) does more rendering? Wouldn't the render be pulling from (A) to post to (B), and would that be too much for (A) and lead to dropped frames?

If it worked I could clean out the older capture scratch files to keep (A) light and archive from (B) to keep (B) light.

Let's start with the idea that software managed RAIDs are not a good idea. Hardware managed RAIDs are expensive for a (good) reason. A RAID 0 will NOT help with rendering. Rendering is a frame by frame processor intensive activity- it doesn't take much to write one-frame at a time. You can decrease rendering times on a Mac by using Compressor and multiple Macs and essentially creating a render farm.

Today's drives are essentially fast enough to handle most any capture without needing a RAID. In terms of workflow, having something like a MacGurus Burly Bay, with swappable drives is great: each project goes on it's own drive. This allows you to capture to one drive while editing a project on another drive (provided you have the software to allow you to do both). Or, have one computer as a capture station to a single drive and then remove that drive and insert into tower for editing. Bottom line is that solutions are out there but typically, a RAID is NOT a solution. And for what it's worth: I have had software and firmware (but not hardware) RAIDs.

Mike

Robert, you take all the fun out of life :-). Yup, I'm violating several rules with this one; and nope, I wouldn't recommend it to anyone. I'm not using the maxupgrade kit, you are correct that the maxupgrade kit is not kind to the air flow. My design is a little better in that regard. Also, my power supply calculations almost reach the limits of it capabilities.... almost, but it is within the capabilities of the power supply as specified. I'm confident that the good engineers at Apple over designed and under spec'd it. That's how I design anyway. Yup, it pretty much nulls the warranty. I'm a EE in my day job and I could write a book on how customers find creative ways to violate warranties. Please point me to these white papers of which you speak, I've been searching and can not find.

Giroud, I've been running with this for about a year. When I first installed it I partitioned it as 2 RAID 0 drives and ran a number of test rendering files sourced on one drive and targeted to the other. Didn't really see a difference in render times from a single drive configuration. Doesn't make sense to me either, but that what I found to be the case. As Mike mentioned it probably won't help render times.

Mike, No need for RAIDs??? Not sure I agree with that. I suppose if all you do is ingest via firewire then you would be correct. However, I wasn't able to capture uncompresses 1080i until I started using a RAID.

Dana, my apologies for taking your post off topic. I believe your solution will work just fine.

No worries, Mark. This is all great stuff.

>You can decrease rendering times on a Mac by using Compressor and multiple Macs and essentially creating a render farm.

I want a render farm! Would you post a new thread on how you can utilize other networked comps using compressor? That is too cool.


MacGurus Burly Bay sounds great, but swapping drives continually concerns me per wear and tear. It seems a violent process to do one time, let alone several times each week. It seems safer to switch firewire plugs. Even better would be a switcher hub, but I haven't seen any.

Render farm: http://manuals.info.apple.com/en_US/...sing_Setup.pdf

In regards to my comment about needing a RAID, the OP was capturing HDV, in which case a RAID isn't needed. I didn't mean it to say that a RAID was never required.

Mike

My second WD 750g external's power died, so I'm about to rip it apart also and slide it in the third internal dock next to the other WD 750g former external. So now I can RAID them in theory, but I don't know how to set up a hardware RAID if I need to buy anything.

I bought a Blu-Ray burner. Now I can physically backup at .48 per GB. To buy and back up to externals cost .21 per GB, so I guess I'll still be buying externals.

Workflow:
1. Capture on second Mac to an external so I can edit on the quad.
2. Connect/access those captured filesover the ethernet to my quad.
3. Edit/render to the two internal 750s.
4. Burn to BR RW. (test)
5. Burn to client's inkjet BR.
6. Fill up internal 750s with render files and eventually dump them onto another external.

Thoughts?

Dana,

This thread got me thinking about hardware raids about a month back and since then I've been researching what is necessary. For starters you need a good controller... Apple makes their own ($$$). CalDigit looks to have a good one with both interanal and external drive support, but I don't like the fact that it's external ports will only work with CalDigit enclosures. Then there are cards from Highpoint, Atto, Acera, and a few others, you'll have to do your own dd to figure out which is best for you, but I would sugest that you stick to the ones that use the Intel 800 MHz IOP or faster. Happy hunting.

Mark

I guess the only option w/o spending $$$ is to fill up one, then fill up the other.

What did you think of the workflow?

Workflow looks ok... but then what do I know :-) Only thing that comes to mind would be to make sure your ethernet hardware is all 1000baseT capable otherwise transfer times will be slow.. about 1.5 min/GB on 100BaseT; a full 60 min HDV tape would take about 18 minutes to move from one mac to the other, using 1000baseT this would only be about a minute.

Having not made the plunge to blu yet, not sure I can comment on that part of it. How much data can you put on a disc???

Be wary of inexpensive hardware RAIDs such as the Highpoint and I even think the Cal-Digit.... these are Fireware RAIDs, not true hardware RAIDs. Hence the price differences. I have been told that the biggest disadvantage to a Firmware RAID is an OS update can knock you out. Or a Firmware update could potentially do the dame.

A couple of year ago I went through all of this: wanted faster throughout, data protection and so on. At one time, I had to Raptors RAID 0 booted off an internal card. It's just such a waste of time. Single drives are far easier to manage, back up, and store as archives. The only time I noticed a difference using my Raptors was when I launched an application. After that I couldn't tell a thing. Simply put, to do it right (and why do it any other way) a hardware RAID is expensive.

Mike

Mike, I would agree... hardware raids can be expensive. However, you should update yourself a little; most of the better raid cards coming from CalDigit, Highpoint, etc these days are true hardware raids with SATA and SCSI interfaces not firewire. They use the same IO controllers as the "big boys". You are correct that running an app off a raid won't buy you much, but if used as intended, as a scratch, capture, or render disk you should really notice a difference. And prices are constantly dropping; the current estimate for pre-configured systems is about $1000 per terabyte. If you have more time than money (like me) and the ability to research and understand the technology you can DIY for much less.

I see that Caldigit is offering a hardware RAID controller for $535 (I didn't see a Highpoint Mac compatible hardware RAID controller): CalDigit | RAID Card | 760600 | B&H Photo Video

My post should have said FIRMWARE, instead of Firewire. My apologies! A standard PCI SATA card isn't going to offer a hardware RAID. Looking at the Caldigit towers and I didn't see where any of them offered a hardware RAID until you get up to the HDPro ($$$)!

Mike

Mike, correct me if I'm wrong but the

Highpoint Rocketraid 3522: HighPoint Technologies, Inc -Mac Support
The CalDigit Raidcard: CalDigit RAID Card
And the Acrea ARC1620: Areca Technology Corporation

are hardware raid cards.

If you don't consider these to be hardware raid, then what would you consider to be hardware raid???

If it quacks like a duck...
 

Mike, looking back over your last post got me to doing a little more digging...

Both the Caldigit HDPro and the HDOne use the same type of connection: PCI bridge <--> Cable <--> Intel Xscale processor <--> drive array. Caldigit calls both hardware raids. The Caldigit raid card just moves the processor onto the PCIe card so that it's connection looks like: PCIe interface <--> Xscale processor <--> cable <--> drive array. It looks like the HDPro and HDOne just differ in bells and whistles. The raid card most likely uses a slightly less capable processor with slightly less throughput.

The Intel Xscale processor family of I/O processors (Intel® I/O Processors) include the IOP331/2/3 and the IOP341/2/8 processors all of which are designed for raid controllers. Because the IOP331/2 are not "recommended for new designs" you can rule them out. From the Caldigit site I can't narrow it down any more than that for their raid card, however there is a reference on the apple forums states it uses the the IOP331 but I haven't been able to verify this.

Highpoint states that they use the IOP341 for the 3522, and Areca states that they use the IOP348 for the ARC1680x.

The AMUG reviews show a slight performance edge for the Rocketraid 3522 over the ARC-1680x in raid 5 and 6 configurations, however the tests are not done with the same drives. This seems strange as I would think the 1680 should perform better with the IOP348.
AMUG HighPoint RocketRAID 3522 PCIe SATA RAID 6 Controller Review
AMUG Areca ARC-1680x PCIe to SAS RAID Controller Review

From the Caldigit site their card looks to have comparable performance but again it looks like an apples to oranges comparison. The only review I've been able to find with numbers compares it to the Apple raid card which it handily beats. CalDigit SATA RAID card

There are other differences, like battery back up and array managers that might make one better suited for a particular application, however all look to be sufficient as hardware raid controllers.

Mark

Mark,

I think we are in agreement! Although I was going by the Caldigit bullet points where they state the HDPro is 100% hardware RAID and they don't mention this with the HDOne. But I agree that the listed cards are hardware RAID controllers.... but at better than $500 for the card. I just wanted to point out that getting a SATA card in general did not offer a hardware RAID solution!

Mike

Mike,

Actually all this "digging" has been benificial for me. I thought I had it cold, but I've learned a few things along the way. I've enjoyed the conversation and the push to do a little more research along the way.

Mark

I use Adobe Premiere Pro CS3 to capture, and Edit in Final Cut Pro at the same time. Premiere's Captured footage works just fine in FCP and therefore lets me do both at the same time.