Archiving video - best solution needed at DVinfo.net

Go Back   DV Info Net > The Tools of DV and HD Production > Open DV Discussion

Open DV Discussion
For topics which don't fit into any of the other categories.


Reply
 
Thread Tools Search this Thread
Old January 13th, 2011, 08:31 AM   #1
Regular Crew
 
Join Date: Oct 2005
Location: Perth Australia
Posts: 121
Archiving video - best solution needed

I need to know what systems people are using to archive video. Are they storing online, if so, where?
Are they storing on hard drives, if so, using what software (please let me know a range if you know them)

Any advice much appreciated if you can help. thx
Geoff Jak is offline   Reply With Quote
Old January 13th, 2011, 12:21 PM   #2
Regular Crew
 
Join Date: Jan 2007
Location: Brooklyn, NY
Posts: 135
I always liked the Blu-ray option. Get some archival discs, burn the files 25gb at a time, then store in peace.
Casey Krugman is offline   Reply With Quote
Old January 13th, 2011, 04:51 PM   #3
Regular Crew
 
Join Date: Oct 2005
Location: Perth Australia
Posts: 121
Thx Casey

I've never used Blue Ray but dont discard the idea....

What about a simple terra SATA II drive. That would be the most cost effective medium wouldn't it?

Is anyone dumping footage online?

And what about software to manage the scenes. If I want to catalogue all scenes - eg, landscsapes, cityscapes, cars, trucks ...is there software I can use to put things in order.
Thx
Geoff Jak is offline   Reply With Quote
Old January 13th, 2011, 08:48 PM   #4
Inner Circle
 
Join Date: Feb 2007
Location: Tucson AZ
Posts: 2,207
Just a quick question - how do you define an archive?

In other words, are you expecting it to retain it's contents for a couple of weeks, a couple of months, a couple of years, a decade or more, etc? It makes a difference

Generally for long term storage nothing you can burn on a computer is truly archival, nor is any hard drive. Hard drives are designed to spin, not sit on a shelf for a year or two or more, and despite the talk about archival DVD's they aren't really to be trusted for long term storage.

LTO tape is probably technically the best medium around for long term storage, as it IS designed to sit around for long periods of time, but it isn't particularly cheap, and not particularly user friendly, although there have been some developments recently that are addressing that.

Maybe the real key is that for an archive to be secure, it has to be accessed and checked regularly and there needs to be a backup archive or two from which you can recover problems with "bit rot"/file corruption, which will happen.

But it does really depend on your specific needs.

Kind of a glum picture I'm afraid, although we're working on it. (I'm really in the storage business and we're very seriously looking at ways to structure so called "archive appliances" that in addition to keeping data safe also provide automated and non-disruptive ways to migrate data to newer technologies as the older technologies age. Which is another requirement of "archive" that isn't often appreciated.)
Jim Andrada is offline   Reply With Quote
Old January 15th, 2011, 10:07 AM   #5
Major Player
 
Join Date: Sep 2003
Location: Rego Park , NYC
Posts: 649
Ever since I got into this Canon DSLR video thing, I've been trying to figure a way to offload and archive the huge amount of files I've generated.

I finally decided to get a eSATA PCI card for my PowerMac G5 and a Kingwin brand eSATA/USB dock from Amazon.

http://www.amazon.com/Kingwin-3-5IN-Esata-Docking-Station/dp/B001HNO3T8/ref=sr_1_1?ie=UTF8&s=electronics&qid=1295107350&sr=8-1
This to me, is an affordable and convenient solution, not to mention how wickedly fast eSATA is.

I will just buy more bare naked drives as I need them in the future.

I have no idea of the durability of long term storage on hard drives, but I can't beat the price.
John C. Chu is offline   Reply With Quote
Old January 16th, 2011, 01:02 AM   #6
Inner Circle
 
Join Date: Feb 2007
Location: Tucson AZ
Posts: 2,207
Yes, price is pretty attractive - until you lose your video one day.

That's the problem in a way, lots of cheap storage available. If you go this route, please make at least two copies and preferably three of each disk and store them in different places. It's not IF you will lose something, but WHEN you will lose it. And occasionally mount the old disks and cross compare the two or three copies. Bit rot happens.
Jim Andrada is offline   Reply With Quote
Old January 16th, 2011, 07:37 AM   #7
Trustee
 
Join Date: Dec 2006
Location: Little Rock
Posts: 1,383
Backing up to raw 1TB drives.
__________________
David W. Jones
www.joneshdfilms.com
David W. Jones is offline   Reply With Quote
Old January 16th, 2011, 04:56 PM   #8
Regular Crew
 
Join Date: Oct 2005
Location: Perth Australia
Posts: 121
Thx to all for your responses.

John C Chu's solution looks a good starting point and maybe the end one too.But what about software to access a particular shot. Is anyone using a program that catalogues each scene so I can find a particular shot when needed.

Hard Disk is the way to go for now - on a big terra drive it seems but I definitely take the point that making two or three copies and checking them periodically is a necessity. We, all of us, really are in a precarious position given the immature stage the archive solutions business is at. There is plenty of room for this business to grow provided it is a cost effective siolution.

Jims comment "Bit rot happens"...is kinda scary, but accurate - just slowly lurking back there is the slimy shadows...beware!
Geoff Jak is offline   Reply With Quote
Old January 16th, 2011, 06:07 PM   #9
Inner Circle
 
Join Date: Feb 2007
Location: Tucson AZ
Posts: 2,207
One thing that IBM is talking about on their LTO tape drives that implement a file system is the possibility of making the drive itself "MXF" aware - the idea would be to recognize MXF files and store the metadata in the directory partition - this would at least allow retrieval by timecode range without having to transfer the whole file to get a section of it. Not sure how you'd do it on HDD though - somehow the user level metadata would have to be kept in a data base of some kind that would reference the file names. Basically logging the shots, which I think most people do already.

Ah yes,"cost effective" - hard to define cost effectiveness. $250k cameras and lenses are certainly "cost effective" for their users or nobody wold buy them. "Cost Effective" for one man or small outfits is another thing altogether.

Let me turn it around and ask what you would consider a cost effective solution?

Just for kicks, assume a solution at price points of say 10TB/50TB/100TB/500TB/1PB

I'd be curious to know what people would consider cost effective at these levels. And what level of storage would be needed.

Or think of it another way - if there were a really good complete solution for archive (and backup while we're at it) out there, would it be reasonable to spend about the same on such a system as on a pro camera/lens combination?
Jim Andrada is offline   Reply With Quote
Old January 16th, 2011, 07:12 PM   #10
Inner Circle
 
Join Date: Oct 2001
Location: Honolulu, HI
Posts: 2,053
I've been using SATA mirrored RAIDs for the past 7 years and haven't lost anything yet.

I'm using Firmtek systems with Hitachi SATA drives.
__________________
Dean Sensui
Exec Producer, Hawaii Goes Fishing
Dean Sensui is offline   Reply With Quote
Old January 16th, 2011, 10:56 PM   #11
Inner Circle
 
Join Date: Feb 2007
Location: Tucson AZ
Posts: 2,207
You're lucky. I know a few people who aren't as lucky. Do you periodically mount the drives and check them for errors - corrupted files, etc?
Jim Andrada is offline   Reply With Quote
Old January 16th, 2011, 11:29 PM   #12
Inner Circle
 
Join Date: Oct 2001
Location: Honolulu, HI
Posts: 2,053
Jim... the RAID software lets me know if there is a problem with one of the drives. If there is, I make yet another backup, replace the bad drive and sync the RAID.

I've done that twice. Perfect recoveries.

Mirrored RAIDs are used for mission-critical operations for everything from banks to air traffic control.
__________________
Dean Sensui
Exec Producer, Hawaii Goes Fishing
Dean Sensui is offline   Reply With Quote
Old January 16th, 2011, 11:54 PM   #13
Inner Circle
 
Join Date: Feb 2007
Location: Tucson AZ
Posts: 2,207
The first example of mirrored RAID I came across was before anyone had thought of the term RAID - the Airlines Control Program developed by IBM in the early sixties maintained mirrored drives across the airline reservation system complex. It worked well then and works well now.

But if drives are stored on the shelf (ie not spinning) the controllers won't detect the out of sync issue and it's possible for them to drop data. If the system is always or regularly powered up, the scan is pretty effective. Unless of course you get a controller failure which wipes out both copies (or the equivalent in a RAID 5 config0. Controller failures do happen (I've seen them fail and corrupt all the members of a RAID array)

More expensive commercial array systems usually have dual controllers to avoid this kind of failure.

You actually made an important comment - ie you referred to making a new backup - typically RAID is thought of as a means of protecting against a single (or more) failure, but it won't protect against losing data intentionally - ie making a human error.

RAID protects against errors in online (ie active) storage but in and of itself it isn't an archiving methodology.

One thing that's happening these days is that drives are getting so big that recovering a bad drive can take many hours - or more, and the exposure to a second (fatal) error increases wit recovery time. Some products like IBM's XIV system (developed by an Israeli company that IBM bought a couple of years back) gets around this by scattering data all around the system so no one drive has a copy of any other singe drive. This allows the recovery to read several drives in parallel and write to several other drives in parallel, thus reducing this kind of exposure.
Jim Andrada is offline   Reply With Quote
Old January 17th, 2011, 09:54 AM   #14
Regular Crew
 
Join Date: Mar 2003
Location: Houston, Texas
Posts: 52
For what it's worth, I save all clips to an external hard drive AND burn them to BD, one copy for my files and another I send to the client. Although most of my prior footage is on BetaSP and i'm sort of new to flash card recording, I have NEVER had a client come back 3 or 5 or 7 years later looking for footage. By that time, most of it has been relegated for demo reel use. After a few years, I send it to my clients and make it their problem. But certainly a good idea to exercise a hard drive every now and then. I do like the idea of LTO tape drives, but seem too pricey right now for me.

Would be interesting to know what NASA uses for image storage, though I expect it to be on the order of a dedicated Cray computer.

My 2 cents only.
Patrick McLoad is offline   Reply With Quote
Old January 17th, 2011, 12:43 PM   #15
Inner Circle
 
Join Date: Feb 2007
Location: Tucson AZ
Posts: 2,207
Interesting question - I don't know what NASA uses. Although a Cray wouldn't be particularly relevant for data storage.

I just saw where the Kennedy archives are now available online in digitized format - they didn't say too much about the system but it was a cooperative effort that included EMC the big storage array maker.

Harvard is in process of digitizing everything in their library system and I know that they keep three copies of all the digitized data in different physical locations. They scan the archives on a regular schedule verifying checksums and when (not if) they detect a file corruption they recover by looking at the other two copies. Not sure of the exact details though.

I guess one of the real questions (which you address) that people have to answer for themselves is how long to retain the archived data - or rather, at what point they wouldn't be impacted by losing it. Clearly something like the Hollywood film archive is intended to be kept forever as are university library and museum collections, presidential archives, exploration data held by the oil companies, etc etc. (There is an archive in Stavanger Norway that has decades of tape cartridges that hold all the data recorded by the North Sea petroleum exploration ships. Oil companies reprocess old tapes with newer algorithms and faster machines, and do indeed discover oil that was missed by earlier analyses,)

But how long do we really need to keep the data around - ie at what rate does its value decrease to the point where losing it isn't a big deal or it isn't worth the cost of keeping the media around anymore. Once you have an answer to this, the archive decision gets simpler - if you think you'll only need it for a few years, then the odds of the media on which you stored it becoming obsolete becomes less important - but if you want to keep it for 10 or 15 years or more, then future proofing becomes important.

One of my Japanese clients had a sudden request from one of their customers for two reel to reel tape drives that had been out of production for - years and years. Apparently the customer had some important data that needed to be accessed and no working drives to access the data with. So my client built two new drives for them pretty much by hand. Would you like to hazard a guess how expensive that was? Don't even ask!

So just how long do you think you want to keep those old (digital) photos of your grandmother before just deciding to throw them out???
Jim Andrada is offline   Reply
Reply

DV Info Net refers all where-to-buy and where-to-rent questions exclusively to these trusted full line dealers and rental houses...

Professional Video
(800) 833-4801
Portland, OR

B&H Photo Video
(866) 521-7381
New York, NY

Z.G.C.
(973) 335-4460
Mountain Lakes, NJ

Abel Cine Tech
(888) 700-4416
N.Y. NY & L.A. CA

Precision Camera
(800) 677-1023
Austin, TX

DV Info Net also encourages you to support local businesses and buy from an authorized dealer in your neighborhood.
  You are here: DV Info Net > The Tools of DV and HD Production > Open DV Discussion

Thread Tools Search this Thread
Search this Thread:

Advanced Search

 



Google
 

All times are GMT -6. The time now is 10:34 AM.


DV Info Net -- Real Names, Real People, Real Info!
1998-2017 The Digital Video Information Network