Best storage solution for wedding videographers
 

Hey guys! First off, I apologize if this matter has been discussed, I briefly browsed the forum to look for it but did not find anything relevant.

Last summer I upgraded my PC to an i7 4790k and 16GB of RAM (for those who don't know already, 16 gigs is NOT enough to work on a lengthy wedding in Premiere, I sometimes get errors that I'm running out of memory) and I also paid about $300 for 3 Western Digital 2TB *Red* HDDs out of which I made a RAID5. It seemed like the best idea to have 4 out of 6TB of space and not losing anything if either HDD failed.

As for Premiere, it's on a SSD along with Windows. Here's my problem.

I'm using the mobo's RAID controller which apparently is producing very slow speeds for my RAID5. Even when I'm copying footage off a memory card, sometimes the transfer speed goes down to 30-40 MB/s even though my card is capable of 60. I noticed that if I turn off buffer flushing in Windows, speed goes up to around 80-90 MB/s, but in the event of a power surge, I'm screwed (data is corrupted/lost).

I would just like to know what you guys use to store a lot of wedding footage (I have filled my 4TB already), if you work directly from the storage HDDs (which I do), and if you think that getting a ~240GB SSD just for my currently active project would be a good idea.

For some reason, as I go past an hour or so in my timeline in Premiere, everything feels sluggish. For instance, if I apply a fast color corrector to a clip, it takes maybe 5-6 seconds to see any modification appear in the monitor. I'm hoping that it's a HDD issue that will be resolved if all my footage is on a SSD.

Any input is greatly appreciated. Thank you!

Re: Best storage solution for wedding videographers
 

you dont say what video card your using
I have a similar system except my processor is i7-4770 CPU @ 3.40GHz i do daily backups to external drives and one internal (slow) green WD drive. Yeah my system could do with more RAM, I dont apply a lot of effects so it chuggs along good enough.
I wouldn't edit from storage at all they are just for storage and nothing else, i eventually archive projects to BD and free up HD space that way. I was looking at getting an SSD which should improve performance.

Re: Best storage solution for wedding videographers
 

Sorry, my video card is a GTX 760 2GB. During those "freezes", that period that I have to wait to see the applied effect to my video - I looked at the GPU usage and it's nothing. I assumed it doesn't play a key role. CPU is barely used, too. That's why I'm looking at my HDDs.

By the way, I recently got a GH4 and started shooting a bit of 4k, especially where low light was an issue (such as some receptions). I'm guessing it would be wiser to get a SSD just for the wedding I'm working on, render the final cut, move it to a storage HDD and then clear the SSD. Rinse and repeat. I know these SSDs can handle 70-80 TB of data traffic so I don't have a problem with replacing it every few years.

Re: Best storage solution for wedding videographers
 

You don't want to work on slow storage drives for editing, you want a drive ONLY for editing. Try another SSD.

As for storage, delete anything you no longer need.

You need a drive for your current video
You need storage for holding pending projects
You need separate storage for archived projects.

For archived I use WD Passports with USB 3 and I detach them when not in use.

Total I have six drives, everything has a backup.

Running raid off of a MOBO controller is not good idea, it's a waste. Just get fast drives and backup everything manually, it's simpler, cheaper and faster.

Also, I suspect the RED drives are no good for raid anyway, but I could be mistaken.

Re: Best storage solution for wedding videographers
 

Thanks for your input Jeff. I'm very nearly about to order a new SSD. Excited about it, too. I shall be sticking with Samsung since the one I have for my OS has been treating me very nicely.

The Red series from WD is specifically made to be used in RAID. That is the only reason why I chose Red instead of Black. If you look on WDs website it even says that the Blacks are not made for RAID purposes, but I did hear of people who run them that way. Then again, I guess you can run any 2 identical HDDs in RAID. It's just that the reds are made to run 24/7 and *should* last longer than a Black. They are also made for use in a NAS.

Re: Best storage solution for wedding videographers
 

My mistake on the Reds, I used to be current on RAID stuff, but I haven't run RAID for two years. I ran it for ten years and am so glad to be done with the controllers and ridiculous boot times.

I ran Seagate Cheetah 15K SAS drives and they were really nice, I do kind of miss the speed, I like them better than my SSDs, my system has never been as stable since I went to SSDs for some reason.

Re: Best storage solution for wedding videographers
 

I have 16GB Ram in my i7-2600K and have no problems with Premiere - and that's a long Catholic wedding filme complete with mass - well over 2 hours on the timeline and 3 or 4 video tracks and the same number of audio tracks

Premiere requires you edit from a drive capable of 7200rpm - I'm not sure your RED drives are up to this, pretty sure they are 5400rpm - I use 2 X Seagate Barracuda drives for directly editing from have had no problems

Pete

Re: Best storage solution for wedding videographers
 

For reliable and fast raid performance you really need to get a dedicated raid card, also red drives are 7200rpm, wd black drives are not designed for raid as they can result in dataloss, you need to get the WD re or WD red (pro) discs who are designed for nas systems.

I don't have a raid system, I use several separate internal and external discs instead, my videodata can be on disc A and when I export out from my NLE this is done to disc B, if I then want to continue working and build a dvd menu from another project I make sure those files are on disc C. As long as you dedicate discs for specific tasks you get the best performance, if you write to disc A and simultaneously try to read another file from that same disc performance drops considerably.

Re: Best storage solution for wedding videographers
 

I have used premier up to cs3 and back then noticed exactly the same problem, the longer my projects where the slower and unstable my pc became, not sure if a SSD will solve your problems as a 7200rpm disc is plenty fast to handle today's codecs, unless you have a very high bitrate one where raid discs are needed. I"m sure if you just get a WD black disc instead of the discs connected to your mobo's RAID controller you wil see a performance gain, your problem is most likely caused by the RED drives you have in raid.

Re: Best storage solution for wedding videographers
 

like you said Noa, the Red is a 7200rpm drive. it's not that slow either, just about 25% slower than a Black. and the Black is not that much cheaper than a 240gb SSD. i'm still leaning towards the SSD, especially considering the write speed when i export a 20gb wedding :)

Re: Best storage solution for wedding videographers
 

Yes, but could it not be that the raid setup you have is causing the problems? Also when comparing price of SSD vs HD, isn't it a fact that you can get much more storing space with HD for the same price? I have also read articles that a SSD is not that suitable to constant write and read data to like you have when dealing with editing data and that it is more likely to fail in a much shorter time period then with a HD.

It also is not sure your export speeds will increase as it will be the processor that is the bottleneck depending which codecs you are dealing with.

Re: Best storage solution for wedding videographers
 

Currently, I use a RAID 0 configured editing based solution, which is a 4TB from G-Tech. I am most likely going to upgrade in the coming year, so that I have a redundancy with a drive running with a Thunderbolt 2 connection to either a Mac Pro or iMac 5K.

Then, the drives are backed up to a Synology 20TB NAS, which runs in RAID 5 and allows for one HDD to fail. I have the industry standard drives inside those (Red), and that NAS is protected via a battery that plugs into the wall, and gives the server around 40 minutes to power down in the event of a power cut - which has happened once thus far.

This was a recommendation by Danny, who used to post regularly here.

Ultimately, we're at a very difficult time with technology vs storage at the moment.

If I can get a 12TB RAID to work alongside my NAS, I will likely rid most of my weddings RAW footage following 2 weeks of delivery.

Re: Best storage solution for wedding videographers
 

Alright, so I received a few ideas from you guys, for which I thank you. Here's what I currently have and what I intend to do based on your feedback:

SSD for OS and Adobe CC
3 x WD Red 2TB in RAID5 (containing both active and archived projects)

What I intend to do:

- SSD for OS and Adobe CC
- 2 x WD Red 2TB in RAID0 or RAID1 (still debating this - I might go for 0 for a few years then switch to 1 for safety. either way, the performance will be greater than that of a single WD Black)
- 1 x WD Red 2TB for EXTERNALLY backing up active / unfinished projects (will buy a USB3 HDD rack for it) - this will be kept in a drawer most of the time
- BUY a 3 or 4TB HDD (thinking about WD Green) to be used as an archive

Thoughts?

Re: Best storage solution for wedding videographers
 

Is the raid on the motherboard?

With regard to your data, work on the basis that any hard disc WILL fail at some point, so make sure you have more than one copy of any work. I personally wouldn't just rely on Raid 1 backups.

You would need to do some testing to see if Raid 0 is of any benefit on your setup, if you are using SATA3 / 6Gbs you many not see enough benefit to offset the worry.

Raid 1 could mean a reduction in performance on your system as everything has to be written twice.

Re: Best storage solution for wedding videographers
 

Yes, the RAID will be managed by the mobo. I had a RAID0 on my previous system before and it was very fast. I've been told that RAID5 is a whole different matter, though, and that reverting to RAID0 or 1 would give me significant performance increases.

I would be relying on the RAID1 as backup for my active projects AS WELL AS the external HDD. So 2 copies of the work that is not done.

I know RAID1 is slower, but I still think it would be much faster than what I have right now. I attached a benchmark that I just ran, I think it's safe to say those are not normal values for Red drives. My 4TB RAID5 is almost full, I am aware that does slow things down, but the scores are still unacceptable.

Re: Best storage solution for wedding videographers
 

Raid 5 was / is a way of pooling discs into a single drive for redundancy, its was never designed for speed.

Re: Best storage solution for wedding videographers
 

Luc, when a hard drive is nearly full it crawls, period, there is nothing to be done about it. If you're using 5400RPM drives and expecting speed, forget it, ain't happening and never will, I don't care what the specs say.

I agree with you Noa, I don't like SSD for edting. It's what I use now, I jumped on the band wagon and sold my 15K SAS drives and have pretty much regretted it, performance wise I've gained nothing. But I do love the quick boot times.

Re: Best storage solution for wedding videographers
 

Jeff, the Red idles at 5400rpm and accelerates to 7200 when needed. It's called "IntelliPower". I looked at benchmarks for both the Red and the Black. The scores are ~800 and ~1000, respectively. So 2 Reds in a RAID0 will outperform a Black by a significant margin (I will benchmark them after formatting).

15k HDDs are not an option for me, I'm only seeing the WD VelociRaptor 10k. Since I have 3 Red drives already and need to spend money on lenses and other things, I need to make do with what I have. I will buy another HDD just for storing things I don't really need, which is why I mentioned the WD Green.

Since the Reds are made for RAID, I'm still very tempted to go for RAID0, even though it's a bit risky. Considering I'll have an external HDD with all my vital data backed up, it doesn't sound too bad.

My former RAID0 made up of 2 Samsung Spinpoints (which are NOT designed for RAID) held for about 2 years, after which I started getting warnings that one of them was faulty. I could still work for a few more months until it became unusable.

I plan to tell Premiere to store all its cache on the RAID0, so that increased write speed in RAID0 will be very welcome.

Re: Best storage solution for wedding videographers
 

You definitely need a dedicated raid controller to gain the best performance and to have better reliability as well, they are not "that" expensive and should give a better performance instead of the onboard raid controller, eventhough raid 0 is not very safe as one failing drive will loose all data but I have been working with one drive only and if that one fails it's also gone. If you would get a separate raid controller for 2 drives in raid 0 and then have a usb 3 drive connected to backup the contents of those 2 drives in raid the max amount of work you might loose is one day if the raid would fail or less depending how often you apply a back up. If you get a backup software solution to first backup the entire drive which will take a long time but after that it can be configured to do incremental backups which run much faster as it only replaces what is changed and adds what had been added.

Re: Best storage solution for wedding videographers
 

Noa, cheapest one I can find in Romania costs $150, the DELL PERC H200, and it says 6Gb/s SAS. I don't have SAS drives.

In addition, when you want to save up for a lens like the Voigtlander (you are very familiar with my obsession), every amount matters. Besides, I was getting somewhere around 200 MB/s write speeds with my former RAID0 setup on that older mobo, I'm now running a Z97 chipset which really shouldn't have a slower RAID controller.

I will copy the sequence file every day, I should be fine.

Re: Best storage solution for wedding videographers
 

I have a 250GB Samsung SSD as my C Drive. All my programs are stored here.
I have 5 or 6 internal WD Black 1TB drives. One them is my Video drive, all my Edius projects are stored in this drive and the rest are used for backups, Data and the likes.
If my video drive gets full and don't want to delete any projects I just replace my Video drive with another Video Drive and work away.

I also have a hot swap bay in my PC were I can add/remove another HDD and use this drive for more external storage of any projects.

I could be wrong but don't think an SSD speeds up any projects on my Edius HD timeline in any way. I didn't buy it for that.
My processor (2700K 3.50GHz 16GB RAM) is by no means fast going by todays CPUs but I'm more than happy with my PC and workflow, and very little or no rendering at all, it has never let me down.

Never work from your storage drive, why would you do this?

Re: Best storage solution for wedding videographers
 

How's that working for you?

For storage alone, fine, but for editing? Not a good choice, IMO. I've had Intellipower drives, green drives, and the lot and they are are not good for anything but storage. Green drives are just the worst. It seems I've had about every type of drive and tried them in many configurations. I began using pricey Adaptec RAID controllers in the 90's, when good controllers cost $2K and more.

Velociraptors are cheap cheap cheap. You do not need RAID, but if you must, run two in a raid 0, the cost is so small.

All onboard raid controllers ARE

Just do your backups

Re: Best storage solution for wedding videographers
 

How's that working for you?

For storage alone, fine, but for editing? Not a good choice, IMO. I've had Intellipower drives, green drives, and the lot and they are are not good for anything but storage. Green drives are just the worst. It seems I've had about every type of drive and tried them in many configurations. I began using pricey Adaptec RAID controllers in the 90's, when good controllers cost $2K and more.

Velociraptors are cheap cheap cheap. Or you could edit just fine off of a WD Black, great drives, but not for RAID.

All onboard raid controllers ARE NOT THE SAME. THEY VARY GREATLY DEPENDING ON THE BOARD. Generally the people that know anything DO NOT use them. Sometimes they can be fine, but for pros that don't want to waste time on the nonsense they do it right and get separate controllers, or they know for a fact that the onboard controller is a good one and they know how to set it up.

Also, if your present drives are nearly full, you cannot complain about their speed, it's just what happens.

Re: Best storage solution for wedding videographers
 

Since the RAID array had 4TB when I created it, I thought I'd have enough space for both active and finished projects. It was never treated as just storage.

My brain feels very confused right now. I am tempted by the Black but I have 3 Reds and I want to make them work. I still believe I can achieve better performance with 2 of them in RAID0 than just working off a single Black. At the same time the thought that the Black has faster access times is annoying (timeline scrubbing?).

I'm almost ready to flip a coin at this point.

Re: Best storage solution for wedding videographers
 

I see, I keep the two apart. You never know what notion a PC will take and corrupt a drive on you.
Don't chance it.

Re: Best storage solution for wedding videographers
 

Update: ordered a WD Green 3TB that's going to be used for storing finished work, pictures, stuff like that. I might get a rack for it and keep it outside my PC since I won't need to access it often.

When the HDD arrives I will copy all my video stuff on it (including active projects), then strip the RAID5 and turn it into RAID0. Will post the benchmark results afterwards, with the drives empty.

Thanks again to everyone who shared their thoughts on this matter. I am actually curious to see how 2 Reds perform together, in their natural habitat, if I may put it that way. I know they're already in RAID, but without a separate raid controller raid5 does more bad than good, apparently.

Re: Best storage solution for wedding videographers
 

Just one last benchmark on the RAID5, after having freed up 300+ GB and disabling write-cache buffer flushing in windows. Will compare this to the RAID0 scores, hopefully tomorrow after my HDD arrives.

It's a clear boost in speed, but still horrid. Those of you with a Black drive, feel free to post your own scores in Crystal Disk Mark. I ran it at 1000MB and 2 passes. Just curious :)

Re: Best storage solution for wedding videographers
 

I just bought a Seagate Archive HDD 8TB drive for archiving as it's about the lowest price per GB of any driver available now & the 8TB means that I have fewer disks to worry about. The design of these drives mean that they are optimised for short bursts of writes & reads not sustained write performance so they are not supposed to be suitable for regular use. So far I have been very pleased, I will be getting one each for the Mac Pros my wife & I own as the capacity will be fantastic for time Machine backups.

The drive cost me £185 ($288/€264) delivered from Amazon including VAT sales tax.

Re: Best storage solution for wedding videographers
 

That's very interesting Nigel... good find! :) I'd not even heard of Archive HDD's like this until now. I looked this up on Seagates website and looked at the specs data sheet. They show the drive is rated to run 24x7 or 8760 hours per year.

What I would like to understand better is how they calculate their MTBF (mean time between failure) numbers? They have MTBF for thie drive listed as 800,000 hours or in normal terms, approximately 91 Years! So obviously this is not real world actual perfomance data... so how do they figure it out?

And if these things really do last that long.. then why only a 3 year warranty? Why not a 50 year or Lifetime warranty? That of course is a rhetorical question... being a tech guy, I know better but if these drives are really that reliable, I'm seriously going to look into buying a few for my own data backups at well.

Re: Best storage solution for wedding videographers
 

OMG, had not idea there are drives available in this size! I'm so out of it!

I saw today that a 10GB drive is going to be released, if it hasn't been already. Amazing.

Re: Best storage solution for wedding videographers
 

goodbye RAID5, hello RAID0! can't wait to edit off of these drives now!

Re: Best storage solution for wedding videographers
 

The price/performance/capacity ratio of a "old" harddisc cannot be beaten, you can get a 6TB disc for 300 euro while a 1T SSD will set you back 500 euro.

Put a few of these large HD in a raid and that will be all the speed you need to edit, a SSD will not give you any performance gains in that respect because the processor is always the bottleneck, it just is not fast enough to be limited by the HD speed.
SSD is only useful for your OS and programs to decrease startup time, I rather spend my money on lots of harddisc space and that has become very cheap these days.

Re: Best storage solution for wedding videographers
 

But why would you edit on a huge drive designed for storage? These are not designed for performance. It makes no sense. And you have no backup.

Ideally an editor should buy two of these for storage/archiving, the second drive being a copy of the first.

Then you buy a FAST drive for editing ONLY, not storage.

This is not rocket science folks.

How I do it:

1 fast drive for OS

2 large drives for storage and archiving, one drive is a copy of the other. This is where you put your video files when they come off of the camera and store them after your edit is done.

1 smaller, faster drive for editing only. While editing you always save your project in three places each time you save it. You save your work on your edit drive, and once on each storage drive.

At the end of your edit, you delete the edit drive, and leave the other copies on your storage drives. Then you move your next wedding or project onto the edit drive and begin again.

Re: Best storage solution for wedding videographers
 

To me a regular HDD that can do ~150 MB/s is not really a storage drive. And when you put 2 of those together in RAID0 you pretty much beat any other single 7200rpm HDD, at least when it comes to transfer speeds.

I do have a back-up on another, external WD Red for which I bought a USB3 rack. This setup feels pretty solid to me for now :)

Re: Best storage solution for wedding videographers
 

Here's what I do for FCPX on an iMac:

- SSD system drive

- project and media on 12TB eSATA 4-Bay OWC RAID5 (~250MB/s Read/Write)

- regular backup via Carbon Copy Cloner to a pair of cheap 5TB Toshiba drives running as a RAID0 (10TB total) via OSX software RAID. Taped them together for easy handling. Photo attached of this monstrosity ;)

- for more data intensive projects, I use a Thunderbolt RAID5 that reaches ~450MB/s, but for most stuff the cheaper eSATA and USB connections are more than fine.

Re: Best storage solution for wedding videographers
 

Well I know everyone is not condoning online cloud storage but we do use it at the studio for piece of mind.

It was a pain to get the initial Terabytes uploaded, but now it's a no brainier for security. However we do have a pretty fast 25mb upload speed. It takes 1 day to upload a 130gb wedding/event while also backing up to our local 32tb Drobo with dual disk redundancy.

If there is a problem they will expedite ship hard drives to replace the data instead of downloading.

Steve

Re: Best storage solution for wedding videographers
 

Any thoughts on unRAID, FlexRAID and Window's Storage Spaces?

I try to consume as much as possible on this as the needs in storage space continuously increasing....