Any way to break up P2 data for archiving? at DVinfo.net

Benjamin Hill · October 11th, 2006, 03:46 PM

I'm new with the HVX, shooting on 8GB cards and want a reliable way to back my data up on plastic. I have been advised by a friend (who has a CD & DVD replication company) that Double-layer 8.5 GB DVD-Rs are nowhere near as stable or reliable as 4.7GB DVD-Rs. Plus they take forever to burn...

Does anyone know of a software or utility for OS X that can break up the contents of an 8GB card so I can burn it to 2 regular 4.7GB DVDs? (And re-assemble the data later of course)? Since it is essentially just data?

Robert Lane · October 11th, 2006, 03:52 PM

There's no reliable method I'm aware of to break up a contiguous file and re-assemble later. And from my many years of dealing with digital video you really wouldn't want to try.

Archiving methods for P2 have been one of the more hotly debated topics but in all respects, using external HDD's are your best and most reliable/cost effective method.

Dedicate a new HDD in an external enclosure that only gets turned on when it's time for saving new clips - strictly for the task of archiving - and you'll do fine.

Benjamin Hill · October 11th, 2006, 04:32 PM

Thanks for sharing your opinion. HDD's are what I've been told and I'm using that method, but I still want to have it on a 2nd medium. I've had HDD's of all kinds konk out on me. I like the idea of tangible media, and the simplicity and economy of data DVDs. To me it's very much worth exploring. (though I almost wish I had gotten 4Gb cards at this point)

Chris Barcellos · October 11th, 2006, 04:57 PM

Tape Drive a possibility ????

Benjamin Hill · October 11th, 2006, 05:15 PM

That's what I was wondering as well, but I don't know anything about tape backup (i.e., overhead, time, size limitations, etc)

Dan Brockett · October 11th, 2006, 09:52 PM

Hi:

We have been archiving all of our massive amounts of P2 shoots onto both DLTs and DVCProHD tape. The DLT method sucks as far as we're concerned, it's way too slow and slow to unarchive. The DLT drive also cost $8,000.00.

Archiving to DVCProHD has been the best alternative so far but it takes a $25,000.00 deck and is in real time and is therefore also too time consuming.

We do not trust hard drives for archival usage. They are okay for short term but for long term, way too fragile. We have a vault. We have a vault staff. If one of the producers asks for a certain archived hdd from the vault, what if the staffer goes to get the drive, the accidentally drops it? Whoops, no more project. Too risky. And doing double hard drive backups erase all of the advatages of hdd.

Nope, they still don't quite have the P2 workflow worked out yet. I love P2 and shoot it every day but I cannot help feeling that I am working with a format that could be a disaster in the long term, in both costs and hassle.

Best,

Dan

Benjamin Hill · October 11th, 2006, 10:05 PM

Thanks for sharing your experiences Dan. Neither the DLT nor the real-time HD dubs sound too appealing, and I share your enthusiasm & ambivalence about the P2 format. But now I am even more motivated to find out how to magically split & reassemble these 8GB files...

Chris Barcellos · October 11th, 2006, 10:16 PM

Benjamin:

Maybe the Womble people are working on something:

http://www.womble.com/

Benjamin Hill · October 12th, 2006, 09:15 AM

The Womble people? That sounds like something from Jim Henson...

Barry Green · October 12th, 2006, 10:32 AM

Going back to the original post question of "how to break up the footage", be aware that the cards already do that. The cards limit the size of any individual file to 4gb (because they use the FAT32 file system, and FAT32 has a maximum file size of 4gb). So if you record for the entire duration of an 8gb card, you'll find that there are actually two files on the card (which get joined together by the camera into one virtual clip).

As to how you could split those off for archiving... well, that would be a little tricky I guess. I mean, the simplest way would be to copy one of the 4gb files out of the VIDEO directory onto one DVD, then delete it off the card, and copy the remainder of the card's contents onto another DVD. Then you'd have to pair those DVDs together and when it comes time to restore the footage you'd reverse the process; restore the DVD with all the directory structures first, then restore that file from the second DVD back into the video directory copied off the first DVD.

Benjamin Hill · October 12th, 2006, 11:12 AM

You're right Barry, I didn't even think of it; thanks for the always-pragmatic advice.

Dean Sensui · October 12th, 2006, 02:28 PM

Regarding the HDD option, if you're going to put it on a hard drive, use a mirrored (level 1) RAID.

It writes simultaneously to two drives.

Advantage: Fast retrieval, easy to archive. If you use a simple disk cataloging program your projects will be easy to find, too.

If one drive dies, the data remains intact on the other drive while the dead drive is replaced. The data then gets re-mirrored. To test this I actually pulled a drive from a RAID pair while data was being written. Of course that scrambled the data on the pulled drive, but the other drive continued to operate normally. After re-formatting the pulled drive, I re-mirrored the data and nothing was lost or corrupted.

This concept has been used for critical data systems including air traffic control and banks.

Using Hitachi Deskstar drives and Firmtek's hot-swap SATA interface I'm able to store terabytes of video for about $1.20 per gigabyte. About the same price as Varicam tape, but without having to spend $25,000 on a deck.

Mattew Love · October 12th, 2006, 04:55 PM

I've been looking at Raid 3, 5, or 6 setups. You're not having to double up on HDD's you're not messing with dozens of DVD's and it's cheaper than Data tapes or a DVCPRO HD deck. What's the downside of this approach?

Benjamin Hill · October 12th, 2006, 06:55 PM

Good question- well how much are you looking at for a RAID 3,5 or 6 setup?

Mattew Love · October 12th, 2006, 07:07 PM

Well, you have your big name offering from G-Technology in the form of the G-Raid pro. This runs $1800 for 1000 gigs of usable space in a Raid 3 setup.

http://www.g-technology.com/Products/G-RAIDpro.cfm

Then there is the WiebeTech casing with no drives. Heard good things about them and they add a eSata connection which is nice and their set up is Raid 5. $1700 to add your own drives $2364 with 1000 gigs of usable space.

http://www.wiebetech.com/products/rt5.php

Last but not least is an offering I found at Firewire Depot. Don't know anything about the quality of their product, but they offer an empty 5 slot case with Raid 6 and an eSata, FW 800, USB 2.0 connection set for $1250. I figure I can buy 5 320 gig Seagate drives for $500 and for $1750 have a nice 1280 gigs of usable space. Not bad huh? Only wish I could find someone to vouch for their quality.

http://fwdepot.com/thestore/product_...oducts_id/1621

*all prices are listed as from mfr, Seagate drive prices I pulled from Newegg*

October 11th, 2006, 03:46 PM	#1
Benjamin Hill Major Player Join Date: Nov 2005 Location: Iowa City, Iowa Posts: 670	Any way to break up P2 data for archiving? I'm new with the HVX, shooting on 8GB cards and want a reliable way to back my data up on plastic. I have been advised by a friend (who has a CD & DVD replication company) that Double-layer 8.5 GB DVD-Rs are nowhere near as stable or reliable as 4.7GB DVD-Rs. Plus they take forever to burn... Does anyone know of a software or utility for OS X that can break up the contents of an 8GB card so I can burn it to 2 regular 4.7GB DVDs? (And re-assemble the data later of course)? Since it is essentially just data? __________________ youtube.com/benhillmedia linkedin.com/in/benhillmedia

October 11th, 2006, 03:52 PM	#2
Robert Lane Go Go Godzilla Join Date: Nov 2004 Location: Scottsdale, AZ USA Posts: 2,788 Images: 15	There's no reliable method I'm aware of to break up a contiguous file and re-assemble later. And from my many years of dealing with digital video you really wouldn't want to try. Archiving methods for P2 have been one of the more hotly debated topics but in all respects, using external HDD's are your best and most reliable/cost effective method. Dedicate a new HDD in an external enclosure that only gets turned on when it's time for saving new clips - strictly for the task of archiving - and you'll do fine. __________________ Robert Lane Producer/Creator - Bike Pilots TV

October 11th, 2006, 04:32 PM	#3
Benjamin Hill Major Player Join Date: Nov 2005 Location: Iowa City, Iowa Posts: 670	Thanks for sharing your opinion. HDD's are what I've been told and I'm using that method, but I still want to have it on a 2nd medium. I've had HDD's of all kinds konk out on me. I like the idea of tangible media, and the simplicity and economy of data DVDs. To me it's very much worth exploring. (though I almost wish I had gotten 4Gb cards at this point) __________________ youtube.com/benhillmedia linkedin.com/in/benhillmedia

October 11th, 2006, 04:57 PM	#4
Chris Barcellos Inner Circle Join Date: Nov 2005 Location: Elk Grove CA Posts: 6,838	Tape Drive a possibility ???? __________________ Chris J. Barcellos

October 11th, 2006, 05:15 PM	#5
Benjamin Hill Major Player Join Date: Nov 2005 Location: Iowa City, Iowa Posts: 670	That's what I was wondering as well, but I don't know anything about tape backup (i.e., overhead, time, size limitations, etc) __________________ youtube.com/benhillmedia linkedin.com/in/benhillmedia

October 11th, 2006, 09:52 PM	#6
Dan Brockett Inner Circle Join Date: Feb 2006 Location: Los Angeles, California Posts: 2,109	Reply Hi: We have been archiving all of our massive amounts of P2 shoots onto both DLTs and DVCProHD tape. The DLT method sucks as far as we're concerned, it's way too slow and slow to unarchive. The DLT drive also cost $8,000.00. Archiving to DVCProHD has been the best alternative so far but it takes a $25,000.00 deck and is in real time and is therefore also too time consuming. We do not trust hard drives for archival usage. They are okay for short term but for long term, way too fragile. We have a vault. We have a vault staff. If one of the producers asks for a certain archived hdd from the vault, what if the staffer goes to get the drive, the accidentally drops it? Whoops, no more project. Too risky. And doing double hard drive backups erase all of the advatages of hdd. Nope, they still don't quite have the P2 workflow worked out yet. I love P2 and shoot it every day but I cannot help feeling that I am working with a format that could be a disaster in the long term, in both costs and hassle. Best, Dan

October 11th, 2006, 10:05 PM	#7
Benjamin Hill Major Player Join Date: Nov 2005 Location: Iowa City, Iowa Posts: 670	Thanks for sharing your experiences Dan. Neither the DLT nor the real-time HD dubs sound too appealing, and I share your enthusiasm & ambivalence about the P2 format. But now I am even more motivated to find out how to magically split & reassemble these 8GB files... __________________ youtube.com/benhillmedia linkedin.com/in/benhillmedia

October 11th, 2006, 10:16 PM	#8
Chris Barcellos Inner Circle Join Date: Nov 2005 Location: Elk Grove CA Posts: 6,838	Benjamin: Maybe the Womble people are working on something: http://www.womble.com/ __________________ Chris J. Barcellos

October 12th, 2006, 09:15 AM	#9
Benjamin Hill Major Player Join Date: Nov 2005 Location: Iowa City, Iowa Posts: 670	The Womble people? That sounds like something from Jim Henson... __________________ youtube.com/benhillmedia linkedin.com/in/benhillmedia

October 12th, 2006, 10:32 AM	#10
Barry Green Barry Wan Kenobi Join Date: Jul 2003 Location: North Carolina Posts: 3,863	Going back to the original post question of "how to break up the footage", be aware that the cards already do that. The cards limit the size of any individual file to 4gb (because they use the FAT32 file system, and FAT32 has a maximum file size of 4gb). So if you record for the entire duration of an 8gb card, you'll find that there are actually two files on the card (which get joined together by the camera into one virtual clip). As to how you could split those off for archiving... well, that would be a little tricky I guess. I mean, the simplest way would be to copy one of the 4gb files out of the VIDEO directory onto one DVD, then delete it off the card, and copy the remainder of the card's contents onto another DVD. Then you'd have to pair those DVDs together and when it comes time to restore the footage you'd reverse the process; restore the DVD with all the directory structures first, then restore that file from the second DVD back into the video directory copied off the first DVD.

October 12th, 2006, 11:12 AM	#11
Benjamin Hill Major Player Join Date: Nov 2005 Location: Iowa City, Iowa Posts: 670	You're right Barry, I didn't even think of it; thanks for the always-pragmatic advice. __________________ youtube.com/benhillmedia linkedin.com/in/benhillmedia

October 12th, 2006, 02:28 PM	#12
Dean Sensui Inner Circle Join Date: Oct 2001 Location: Honolulu, HI Posts: 2,054	Regarding the HDD option, if you're going to put it on a hard drive, use a mirrored (level 1) RAID. It writes simultaneously to two drives. Advantage: Fast retrieval, easy to archive. If you use a simple disk cataloging program your projects will be easy to find, too. If one drive dies, the data remains intact on the other drive while the dead drive is replaced. The data then gets re-mirrored. To test this I actually pulled a drive from a RAID pair while data was being written. Of course that scrambled the data on the pulled drive, but the other drive continued to operate normally. After re-formatting the pulled drive, I re-mirrored the data and nothing was lost or corrupted. This concept has been used for critical data systems including air traffic control and banks. Using Hitachi Deskstar drives and Firmtek's hot-swap SATA interface I'm able to store terabytes of video for about $1.20 per gigabyte. About the same price as Varicam tape, but without having to spend $25,000 on a deck. __________________ Dean Sensui Exec Producer, Hawaii Goes Fishing

October 12th, 2006, 04:55 PM	#13
Mattew Love New Boot Join Date: Aug 2005 Posts: 11	I've been looking at Raid 3, 5, or 6 setups. You're not having to double up on HDD's you're not messing with dozens of DVD's and it's cheaper than Data tapes or a DVCPRO HD deck. What's the downside of this approach?

October 12th, 2006, 06:55 PM	#14
Benjamin Hill Major Player Join Date: Nov 2005 Location: Iowa City, Iowa Posts: 670	Good question- well how much are you looking at for a RAID 3,5 or 6 setup? __________________ youtube.com/benhillmedia linkedin.com/in/benhillmedia

October 12th, 2006, 07:07 PM	#15
Mattew Love New Boot Join Date: Aug 2005 Posts: 11	Well, you have your big name offering from G-Technology in the form of the G-Raid pro. This runs $1800 for 1000 gigs of usable space in a Raid 3 setup. http://www.g-technology.com/Products/G-RAIDpro.cfm Then there is the WiebeTech casing with no drives. Heard good things about them and they add a eSata connection which is nice and their set up is Raid 5. $1700 to add your own drives $2364 with 1000 gigs of usable space. http://www.wiebetech.com/products/rt5.php Last but not least is an offering I found at Firewire Depot. Don't know anything about the quality of their product, but they offer an empty 5 slot case with Raid 6 and an eSata, FW 800, USB 2.0 connection set for $1250. I figure I can buy 5 320 gig Seagate drives for $500 and for $1750 have a nice 1280 gigs of usable space. Not bad huh? Only wish I could find someone to vouch for their quality. http://fwdepot.com/thestore/product_...oducts_id/1621 all prices are listed as from mfr, Seagate drive prices I pulled from Newegg