DV Info Net

DV Info Net (https://www.dvinfo.net/forum/)
-   Sony XDCAM EX Pro Handhelds (https://www.dvinfo.net/forum/sony-xdcam-ex-pro-handhelds/)
-   -   RAID Archive - Handling the 2TB size limit (https://www.dvinfo.net/forum/sony-xdcam-ex-pro-handhelds/450009-raid-archive-handling-2tb-size-limit.html)

Steve Gibbons September 28th, 2009 01:11 PM

RAID Archive - Handling the 2TB size limit
 
For those of you archiving your EX media to large RAID setups...

How are you handling the single volume size limit of 2TB? We would like all of our video archive on a single drive but this limit poses a problem for us.

Any comments would be appreciated.

Brooks Graham September 28th, 2009 02:45 PM

I have a Mac with a single 16TB volume on a DroboPro, so I'm assuming you're on a Microsoft computer. Solution: get a Mac. ;-)

The second thing I'd say is perhaps you should seriously reconsider using "hard drive" and "archive" in the same sentence. I think people misuse the word "archive" when they mean "near-line" or perhaps "backup". This has been beaten to death here, but IMHO using spinning rust as an archiving medium is like writing on rice paper outdoors in monsoon season. Bad idea jeans.

Tape such as LTO and (some) optical are the only media suitable for actual archiving. Hard drives are not, nor is any flash memory device.

Marc Myers September 28th, 2009 02:56 PM

My understanding is the issue is the sector size. There's a limit to the number of sectors, not the absolute size of the array. When you set up your array, make the sector size as large as possible. For most tasks, smaller sector size is more efficient since if even on byte is occupied, that sector is considered used. Leads to lots of wasted space if one has many small files. Withe video this is never a problem.

Brooks Graham September 28th, 2009 03:23 PM

The XP partition table size limit is 2TB. There are hacks that allow a partition table entry to get expanded beyond that, but for this use, do not do that. You want to maintain the highest level of compatibility, not lower it. For the record, NTFS has a filesystem size limit measured in exabytes (insanely large by today's standards).

If you switch to Vista or beyond, it can use a different partition table scheme and support much larger volumes. And of course, Macs have long supported volumes beyond 2TB.

And to be very, very clear on this, changing hard drive sector sizes is not a good thing, and in most cases impossible. Hard drive sector size is 512 bytes. With some filesystems, you can change the filesystem block allocation size, but sector sizes are fixed at 512 bytes with hard drives. Again, twiddling any of these things is a bad idea if you want to someday read that filesystem on some other, foreign system.

Steve Kalle September 29th, 2009 01:32 AM

The solution is very simple - use GPT partitions instead of MBT.

However, if you are using Raid, then you also have a Raid controller. I cannot stress this enough: you MUST use a hardware Raid controller for important data, which means not using the onboard Raid and/or software Raid. Most hardware Raid controllers provide a "Carve" feature which allows you to carve the volume into 2TB volumes for legacy support generally.

Furthermore, archiving to ANY Raid array is not a great plan for one important reason: the Raid controller which created that array limits what Raid controllers can read that array in the future should there be a problem with the original controller. I learned this the hard way with a 3ware controller having issues and a $700 Adaptec couldn't read the Raid 5 array and I had to get another 3ware controller for it to work.

Taky Cheung September 29th, 2009 02:23 AM

MBR (Master Boot Record) has the limitation of 2TB. So like Steve said, format your partiton as GPT (GUID Partition Table) allow format the drive over 2TB limit.

I'm using RocketRaid 2640X4 a PCI-e x4 raid card for a 4x 1TB Raid-0 setup. Works very well.

Steve Gibbons September 29th, 2009 11:15 AM

Quote:

Originally Posted by Brooks Graham (Post 1410303)
I have a Mac with a single 16TB volume on a DroboPro, so I'm assuming you're on a Microsoft computer. Solution: get a Mac. ;-)

The second thing I'd say is perhaps you should seriously reconsider using "hard drive" and "archive" in the same sentence. I think people misuse the word "archive" when they mean "near-line" or perhaps "backup". This has been beaten to death here, but IMHO using spinning rust as an archiving medium is like writing on rice paper outdoors in monsoon season. Bad idea jeans.

Tape such as LTO and (some) optical are the only media suitable for actual archiving. Hard drives are not, nor is any flash memory device.

I'm actually very much an advocate of LTO-4 as a solid archive solution (see some of my other posts about Cache-A products). And we are in fact using it in our facility - but more so as part of our full office backups. Our XDCAM-EX "folder" is part of that backup for now but only in an "emergency contingency" role as opposed to a library one.

LTO-4 is well suited as a long-term archive (eg. 1 year+) but less so for working with media on a near or mid-term basis in a collaborative environment. We need media hot in a "near line" sense but given our XDCAM-EX archive size, the 2TB mark will be passed by late winter. Hence why we are looking at other large single drive (or volume if you prefer) RAID solutions that can act in this role.

The Drobo certainly caught our eye but had some limitations within a traditional LAN.

SG

Mitchell Skurnik September 29th, 2009 03:10 PM

Rebuilding anything above 2TB is going to take a long time. If you have multiple drive failures you are going to have to wait hours or days to recover your data.

Tuy Le October 1st, 2009 01:34 PM

I'm running win7 on PC and it's 3TB (1.5 for each drive)


All times are GMT -6. The time now is 03:49 PM.

DV Info Net -- Real Names, Real People, Real Info!
1998-2024 The Digital Video Information Network