RAID Archive - Handling the 2TB size limit at DVinfo.net

Go Back   DV Info Net > Sony XAVC / XDCAM / NXCAM / AVCHD / HDV / DV Camera Systems > Sony XDCAM EX Pro Handhelds

Sony XDCAM EX Pro Handhelds
Sony PMW-300, PXW-X200, PXW-X180 (back to EX3 & EX1) recording to SxS flash memory.


Reply
 
Thread Tools Search this Thread
Old September 28th, 2009, 02:11 PM   #1
Regular Crew
 
Join Date: Mar 2008
Location: Bridgenorth
Posts: 116
RAID Archive - Handling the 2TB size limit

For those of you archiving your EX media to large RAID setups...

How are you handling the single volume size limit of 2TB? We would like all of our video archive on a single drive but this limit poses a problem for us.

Any comments would be appreciated.
Steve Gibbons is offline   Reply With Quote
Old September 28th, 2009, 03:45 PM   #2
Regular Crew
 
Join Date: Nov 2008
Location: San Francisco, CA
Posts: 121
I have a Mac with a single 16TB volume on a DroboPro, so I'm assuming you're on a Microsoft computer. Solution: get a Mac. ;-)

The second thing I'd say is perhaps you should seriously reconsider using "hard drive" and "archive" in the same sentence. I think people misuse the word "archive" when they mean "near-line" or perhaps "backup". This has been beaten to death here, but IMHO using spinning rust as an archiving medium is like writing on rice paper outdoors in monsoon season. Bad idea jeans.

Tape such as LTO and (some) optical are the only media suitable for actual archiving. Hard drives are not, nor is any flash memory device.
__________________
EX3, Z1U, FX1, D1X / NeXTstation Turbo Color / FCS3
Brooks Graham is offline   Reply With Quote
Old September 28th, 2009, 03:56 PM   #3
Regular Crew
 
Join Date: Nov 2008
Location: Long Island, NY and Northeast PA
Posts: 132
My understanding is the issue is the sector size. There's a limit to the number of sectors, not the absolute size of the array. When you set up your array, make the sector size as large as possible. For most tasks, smaller sector size is more efficient since if even on byte is occupied, that sector is considered used. Leads to lots of wasted space if one has many small files. Withe video this is never a problem.
Marc Myers is offline   Reply With Quote
Old September 28th, 2009, 04:23 PM   #4
Regular Crew
 
Join Date: Nov 2008
Location: San Francisco, CA
Posts: 121
The XP partition table size limit is 2TB. There are hacks that allow a partition table entry to get expanded beyond that, but for this use, do not do that. You want to maintain the highest level of compatibility, not lower it. For the record, NTFS has a filesystem size limit measured in exabytes (insanely large by today's standards).

If you switch to Vista or beyond, it can use a different partition table scheme and support much larger volumes. And of course, Macs have long supported volumes beyond 2TB.

And to be very, very clear on this, changing hard drive sector sizes is not a good thing, and in most cases impossible. Hard drive sector size is 512 bytes. With some filesystems, you can change the filesystem block allocation size, but sector sizes are fixed at 512 bytes with hard drives. Again, twiddling any of these things is a bad idea if you want to someday read that filesystem on some other, foreign system.
__________________
EX3, Z1U, FX1, D1X / NeXTstation Turbo Color / FCS3
Brooks Graham is offline   Reply With Quote
Old September 29th, 2009, 02:32 AM   #5
Trustee
 
Join Date: Aug 2009
Location: Chicago, IL
Posts: 1,554
The solution is very simple - use GPT partitions instead of MBT.

However, if you are using Raid, then you also have a Raid controller. I cannot stress this enough: you MUST use a hardware Raid controller for important data, which means not using the onboard Raid and/or software Raid. Most hardware Raid controllers provide a "Carve" feature which allows you to carve the volume into 2TB volumes for legacy support generally.

Furthermore, archiving to ANY Raid array is not a great plan for one important reason: the Raid controller which created that array limits what Raid controllers can read that array in the future should there be a problem with the original controller. I learned this the hard way with a 3ware controller having issues and a $700 Adaptec couldn't read the Raid 5 array and I had to get another 3ware controller for it to work.
Steve Kalle is offline   Reply With Quote
Old September 29th, 2009, 03:23 AM   #6
Inner Circle
 
Join Date: May 2007
Location: Los Angeles, USA
Posts: 2,109
MBR (Master Boot Record) has the limitation of 2TB. So like Steve said, format your partiton as GPT (GUID Partition Table) allow format the drive over 2TB limit.

I'm using RocketRaid 2640X4 a PCI-e x4 raid card for a 4x 1TB Raid-0 setup. Works very well.
Taky Cheung is offline   Reply With Quote
Old September 29th, 2009, 12:15 PM   #7
Regular Crew
 
Join Date: Mar 2008
Location: Bridgenorth
Posts: 116
Quote:
Originally Posted by Brooks Graham View Post
I have a Mac with a single 16TB volume on a DroboPro, so I'm assuming you're on a Microsoft computer. Solution: get a Mac. ;-)

The second thing I'd say is perhaps you should seriously reconsider using "hard drive" and "archive" in the same sentence. I think people misuse the word "archive" when they mean "near-line" or perhaps "backup". This has been beaten to death here, but IMHO using spinning rust as an archiving medium is like writing on rice paper outdoors in monsoon season. Bad idea jeans.

Tape such as LTO and (some) optical are the only media suitable for actual archiving. Hard drives are not, nor is any flash memory device.
I'm actually very much an advocate of LTO-4 as a solid archive solution (see some of my other posts about Cache-A products). And we are in fact using it in our facility - but more so as part of our full office backups. Our XDCAM-EX "folder" is part of that backup for now but only in an "emergency contingency" role as opposed to a library one.

LTO-4 is well suited as a long-term archive (eg. 1 year+) but less so for working with media on a near or mid-term basis in a collaborative environment. We need media hot in a "near line" sense but given our XDCAM-EX archive size, the 2TB mark will be passed by late winter. Hence why we are looking at other large single drive (or volume if you prefer) RAID solutions that can act in this role.

The Drobo certainly caught our eye but had some limitations within a traditional LAN.

SG
Steve Gibbons is offline   Reply With Quote
Old September 29th, 2009, 04:10 PM   #8
Major Player
 
Join Date: Jan 2008
Location: Rancho Santa Margarita, California
Posts: 201
Rebuilding anything above 2TB is going to take a long time. If you have multiple drive failures you are going to have to wait hours or days to recover your data.
__________________
Mitchell J. Skurnik
http://www.mjcsstudios.com/ - EX1, 4x hoodman 16GB, Libec Tripod, Sony LAV
Mitchell Skurnik is offline   Reply With Quote
Old October 1st, 2009, 02:34 PM   #9
Regular Crew
 
Join Date: Mar 2009
Location: San Jose, CA
Posts: 96
I'm running win7 on PC and it's 3TB (1.5 for each drive)
Tuy Le is offline   Reply
Reply

DV Info Net refers all where-to-buy and where-to-rent questions exclusively to these trusted full line dealers and rental houses...

Professional Video
(800) 833-4801
Portland, OR

B&H Photo Video
(866) 521-7381
New York, NY

Z.G.C.
(973) 335-4460
Mountain Lakes, NJ

Abel Cine Tech
(888) 700-4416
N.Y. NY & L.A. CA

Precision Camera
(800) 677-1023
Austin, TX

DV Info Net also encourages you to support local businesses and buy from an authorized dealer in your neighborhood.
  You are here: DV Info Net > Sony XAVC / XDCAM / NXCAM / AVCHD / HDV / DV Camera Systems > Sony XDCAM EX Pro Handhelds

Thread Tools Search this Thread
Search this Thread:

Advanced Search

 



Google
 

All times are GMT -6. The time now is 01:48 PM.


DV Info Net -- Real Names, Real People, Real Info!
1998-2017 The Digital Video Information Network