Which one would last longer for archiving videos? Store them on DVD or on a Hard drive?

stelios

Let me turn the question around a bit

How long do you think you will store them?

If you're thinking a few months or a year or two, either will probably be OK.

If you're thinking 10 years, I think neither one is reliable.

Generally speaking, I'd go for a hard drive, but be sure to make two copies and also be sure to check the contents periodically. Monthly would be good, but probably nobody will do it. At least check the data once a year.

If you're looking for real long term archive, best thing around is LTO tape, but the drives are $500 - $1000. After that the tapes themselves are competitive with HDD.

Imo, harddrive.

No form of storage lasts really long and it is a lot easier to copy 1 TB of data from one drive to a new one then it is to copy 100 or 200 DVD's ...

In other words you should replace your hard drives every 2-3 years to a new one and copy your data to this new one?

stelios

What I really recommend is a small raid array - some of them automatically scan the data continuously looking for errors which they can recover on the fly.

The problem with storing data on the shelf is that there is no ongoing error checking and recovery.

So one scheme that can work is to make two copies to HDD, then after 12 months or so copy the data to a third HDD and keep cycling around the three drives every year - this way by doing the copy, you have confirmed that all data is readable and recoverable.

Just letting drives sit around on a shelf for a few years is a good way to risk your data.

I'd go hard drive too. Easier to check it every so often and back it up a second or third time.

The problem with storing data on the shelf is that there is no ongoing error checking and recovery.


A good argument for harddrives, having your files "online" you have continuous access to monitor disk status, do background integrity check and recovery as well as fast migration to new media. Consider having to copy 1TB of DVDs to new DVDs compared to copying same size a HDD.

Some say that it's not good to have HDD lying on the shelf, that they should be used to stay operable. I think it's a myth that might once have been true in ancient times, but I doubt it with todays hard drives: The mechanics is in a hermetically closed environment, I don't see why it should not deteriorate faster by not being used.

After all these different media types, storage cost in free fall and data exploding, I don't think there is such a thing as a permanent archive, instead one should plan for continual migration.

BR, Erik

For 20 years I worked in computer storage and I vehemently disagree with those who say hard drive.The truth is that all hard drives fail, full stop. Given enough time, they will. It could be in a year or it could be in 10. The question is, do you want to bet on when? If so, what happens if you're wrong.

Optical is a better solution for archiving. Pardon me for being completely blunt but if you don't know the difference between fault-tolerance, back-up and archiving you don't know enough to intelligently ask this question.

What you are doing with your production content is running a mini-data center. One with a huge amount of data. The best way to find an answer you can comfortably live with is to learn a bit about computer data management. A little research will teach you that different media are appropriate in different stages in data workflow.

If you don't feel like doing the research, go optical, but tape is better.

Any mechanical device tends to work better with use. Any of the experts I've heard recently have stated that bearings losing lubrication over time is the problem with hard drives.

Relying on a cheap hard drive to store your data longer term mightn't be the way to test a myth.

Even if you put all your eggs in a titanium basket, it's still one basket.

Even tapes can go bad, too. Everyone has seen a tape get eaten.

Always make multiple backups, whether it's tape, hard drive or optical.

And no matter what media you choose, be sure there's something that can read it in the future. Because even if the media is intact, it's no good if the system is so proprietary that no one has a working machine that can handle it.

I transferred some old Works documents from my old computer, which could be opened there by Word 2003, onto my new computer and Word 2007 can't open them (even though it's supposed to).

The same thing happened with some old scripts done on Scriptware, that program couldn't open them on my new computer with the same program.

I had to use the old computer to create files of these documents that would work on the new machine.

Ahh yes, the infamous "future proofing" problem. This is a really big deal for archives. Ether you have to periodically migrate from older to newer technology or you have to get the PC related industries to stop making major changes/improvements every year or two. So since that's not going to happen, you're forced into a periodic migration mode.

One of the really good things about LTO is that in order to be certified to use the logo, all drives have to be able to read tapes written on two generation earlier machines and to read and write tapes made on one generation earlier machines. Also, we normally have three generations of drives in current manufacture so you should be pretty well assured that you will be able to buy a new drive that will read your old tapes for at least ten to 15 years after you write them. And service/spare parts will be available for at least 7 to 10 years after final manufacture. So if you write a tape today, you can good confident that you'll be able to read it back 15 to 20 years or more later. The new drives may use different attachment protocols etc than the old one (SCSI vs Fibre Channel vs e-SATA vs SAS vs who knows what) but the data will be readable). A hard drive on a shelf on the other hand offers no guarantee that there will be a system to plug it into even if the data is still good. Heck, we're already running into this with laptops no longer supporting Firewire for camera attach - even if you have a good tape in a working camera, no guarantee you'll be able to attach it to anything you can afford.

Which then sort of begs the question of whether the software that's out there in 20 or 25 years will be able to process the data that you just read back! Will Cineform still be around in 25 years? I hope so. I guess the key is to archive in really standard formats/codecs.

Also - I think we talk about archiving without really defining the length of time we want to be sure the data will be retrievable. The time horizon is really important - if our definition of an archive is something that I can recover a year or two or three later, it's easy. If we're thinking about digital assets with 50 or 100 year or more life spans, it gets to be a different kettle of fish altogether.

I think that a key part of the equation with digital is that you can - and possibly should - re-archive over time, including validation.

Hard drives aren't a bad strategy - if you include double storage and re-archive every four or five years. The nice thing is that sizes keep increasing. You might have 500GB drives one year and 4TB drives down the road. Each time you re-archive, it's possible that you consolidate into fewer and fewer - and less cost-per-byte - devices.

My archive horizon is forever... or at least my lifetime. The horizon of the specific media may be shorter. And that means that maintenance must be part of the plan.

If LTO were random access, I'd be more prone to consider it. Years ago, my wife and I backed up on to tape. It was expensive and time consuming. And thank goodness we never had to access any of it. Because it wasn't random access, we saw it as an emergency recovery system rather than a useful archive.

That was about 15 years ago. Hopefully, the tools have improved since then.

This is really frightening. Everybody now is converting everything into digital form, books, pictures magazines e.t.c. Will my grand children be able to open these archived files? I am now busy taking pictures and videos of my grand children and store them in DVDs and hard drives. Will they be able to open and see these when they become adults? Thank god that I have the old pictures that my parents took when I was a baby. I can still see them with out using any electronic "gadgets" Thank God for the libraries. Books written by Greek, Egyptian and other historians and philosophers can be read today.

stelios

Well, yes, things have changed a bit

Current LTO drives permit drag and drop to copy files to tape. Not quit random access, but I think max high speed search time is on the order of 20 to 30 minutes (depending on the drive.) I'll check the details.

I know what you mean about old fashioned PC tape - when I moved back to the US from Japan I sold my blazingly fast 386 system to a friend and backed everything up to tape to restore on my US system. What a disaster! I stuck with it for easily 10 to 12 days and finally declared a victory when I had restored about 75% of my files.

Yes, Stelios, it is frightening.

And I think the answer is that no, they probably won't be able to see any of your digital archive - UNLESS - you keep it online somewhere in the "cloud" and pay some service to keep it safe and viewable in a browser.

Or print it to paper!!!

Oh yeah - how permanent is computer printout? Really!

Many many many years ago I was very serious about large format black and white photography - to the point that I took classes with people like Minor White and Ansel Adams. And I learned how to process for archival permanence. It was pretty simple - just a matter of chemistry, and getting special storage boxes that were truly acid free. a bit laborious, true, but something you owed to anyone who would purchase one of you prints. Now the idea of permanence is really twisted. A lot of institutions are struggling with the same question of preservation of digital archives as museums and libraries convert their collections to digital. But somehow it will never be the same as paper. I remember in college having a class at Harvard's rare book library - and sitting at a table looking at Isaac Newton's actual handwritten notebooks. Right there on the table in front of me - what a wonderful sensation of historic continuity. The plus of digital - everyone can look at Newton's notebooks without getting up from their computer. The negative - it's sort of a so what experience. Nothing but bits, nothing genuine - only the palest shadows of the palest shadows of the real thing.

Oh well, maybe nobody cares any more.

My system is fairly simple:

I store all source and project files on one HDD at the time of project creation. I can re-render if needed in future. As I fill a drive, I buy an enclosure and label the drive. Cost, about $0.10/GB

I store all rendered files on a separate HDD as I render. If my source drive fails, I have the rendered project. As I fill a drive, I buy an external enclosure and label the drive. Cost, about $0.10/GB

I store my most valuable renders on a SD card and keep separate from my computer (at office) in case of fire or other disaster. My understanding is SD cards are reliable long term, but obviously this hasn't been conclusively proven since they haven't been around for decades yet. Cost, about $1/GB.

I still think inexpensive mirrored hard drives are the way to go. If you periodically consolidate your data, the cost effectiveness just keeps getting better and better over time.

When I started archiving to hard drives, mirrored 100GB drives were around US$50-150. Now I can get mirrored 1TB drives for the same price - and I'm starting to see 2GB drive prices heading in the same direction. With a dual hot-swappable enclosure it's a no brainer.

Whenever I see large drives dropping in price, I buy a set and consolidate all the data, and use the older/smaller drives for current projects. Very cost effective and efficient.

Hard Drives over DVDs. LTOs if the budget allows for it. Other options: Digibeta and HDCAM (SR), etc.

The question really isn't: How long will it last; rather: How long do you want it to last and what price are you willing to pay for it?

Not "how long will it last", but "how long before I need to do another maintenance cycle."

Yes, Stelios, it is frightening.

And I think the answer is that no, they probably won't be able to see any of your digital archive - UNLESS - you keep it online somewhere in the "cloud" and pay some service to keep it safe and viewable in a browser.

Or print it to paper!!!

.

Well Jim well said. I suppose I am going to print those memorable moments for them to see....


stelios

But 19th Century glass plate negatives could be around even longer.

I believe that Mel Gibson's "Apocalypto" was shot on high-definition digital video, using the Panavision Genesis cameras. Does this mean that in 10-20 years it will "loose" it's high definition or unable to play? Interesting thought...

stelios

But 19th Century glass plate negatives could be around even longer.

And look how long some cave paintings have been around.... :-)

Is anyone here archiving to Blu-ray? That's my plan right now.

I believe that Mel Gibson's "Apocalypto" was shot on high-definition digital video, using the Panavision Genesis cameras. Does this mean that in 10-20 years it will "loose" it's high definition or unable to play? Interesting thought...

stelios

The studios commonly make 35mm black & white RGB separations for archiving feature films.

Re Glass plates - I've actually used them. Not for photos, but for astronomical spectrograph analysis.

At school, we had quite a collection of astronomical photos, etc that were taken starting in the mid 1800's (I think the first camera - telescope photo would date to around 1840.)

Glass is great - rather permanent, and stable enough for detailed measurements.

But - it doesn't like being shaken or dropped. Our plate stacks were a 5 story building inside, but isolated from, a regular 3 (or 4??) story building. The stacks had separate foundations and were mounted on springs and shock absorbers, and it was sort of like getting into an elevator to enter the stacks - there was a space of an inch or so between the inner building and the outer building.

Anyhow, I think the point is that archiving ANYTHING is not a toss it on the shelf and forget it operation. It takes management, and effort to keep the assets in as close to perfect condition as possible. Air conditioning and filtration etc for physical assets, continuous checking and copying for digital assets. Probably the easiest thing for individuals would be to store your data in the "cloud" and let them keep it safe (at a price) - for as long as whatever cloud company you use stays in business.

Oh - as I remember reading, cave paintings don't like people breathing on them or they deteriorate.

And look how long some cave paintings have been around.... :-)

And even then, it's a maintenance problem. The most valuable caves are sealed. You can only walk on planks. Etc.

well then I suppose the moral of the story is:
enjoy them today as tomorrow might not be around...

stelios

All archival work requires maintenance, paintings etc are regularly checked my museums etc. Although, It appears that digital media needs more intensive maintenance than other media, which does add a cost that needs to be budgeted for.

I have just had a Hard drive crash on me, it was only 18 months old. Fortunately I only had completed jobs on it, nothing which I could generate more income from, unless a customer wants a single DVD copy. I don't think any media is futureproof. When I first started out in digital imaging I backed everything up on MO disks, which would last for 100+ years, pity the hardware didn't last that long. Now I have disks but no drive to read them with.

At least with photographs you have a hard copy. I guess with film you also have a hard copy, but with digital video you just have data bits.

I have just had a Hard drive crash on me, it was only 18 months old. Fortunately I only had completed jobs on it, nothing which I could generate more income from, unless a customer wants a single DVD copy.

I always make full backups of everything I work on, and assume the striped RAID is going to crash.

At the end of the day current projects get backed up to a separate drive that's taken offline after the copying process is done.

And when a project is completed it gets moved to a mirrored RAID and added to the catalog for future reference. That mirrored RAID is also taken offline and stored on a shelf.

I've worked this way for the past 8 years or more and it's proven safe.

As a rule I also back up material, sometimes, as in the case i mentioned, I do a job and keep it for one or two years and then delete. I am talking about the editable master file. The final DVD files are usually kept ad-infinitum. Or at least a copy of the DVD production is.

I think storage will be the Achilles heel for digital video, until a fail safe long term storage system is introduced.

At the moment I am backing up my video projects on an external WD Elements desktop HD 1 TB as well as on DVD.

When the HD is nearly full I disconnect it from my PC and store it in a cabinet. How long do you think this external drive will last on the shelf?

stelios

"How long do you think this external drive will last on the shelf?"

Not as long as you would think. Hard drives are like car engines, they need to be kept active otherwise they will just lock up. Generally you should spin up a hard drive at least once or twice every six months.

Damned if you do and damned if you don't. Put hard drives on the shelf and lubricants dry out - but probably more deadly is the fact that bits rot away quietly. I've seen a lot of drives that were fine after long periods of inactivity - but also a lot where even though the drive powered up OK the data was corrupted.

In older disk drives there was an issue with the heads coming to rest on the drive surface and surface lubricant wicking into the head disk interface and effectively gluing the heads to the disk surface. It was known as stick-tion and sometimes it was so effective that the head sliders were torn off the suspensions. This was not good.

Modern drives typically "unload" the heads ie pull them clear of the surface on shutdown to prevent this.

Many of the issues of digital are in this report by the Academy of Motion Picture Arts and Sciences

The Digital Dilemma | Science & Technology Council | Academy of Motion Picture Arts & Sciences (http://www.oscars.org/science-technology/council/projects/digitaldilemma/)

The studios commonly make 35mm black & white RGB separations for archiving feature films.

Correction: YCM separations.

After all these info, this weekend I have a lot of precious photos to print...

stelios

... The new drives may use different attachment protocols etc than the old one (SCSI vs Fibre Channel vs e-SATA vs SAS vs who knows what) but the data will be readable). A hard drive on a shelf on the other hand offers no guarantee that there will be a system to plug it into even if the data is still good.

You just say that tape drives suffer from the possible change of attachment protocols, and then you criticize HDD for the exact same problem!

Even if your tapes are readable and data are good, you might be lucky to find a drive that will read them but then there is no guarantee that you can plug the drive into any current system. Same problem as any other storage media suffer from.

BR, Erik

At least with photographs you have a hard copy. I guess with film you also have a hard copy, but with digital video you just have data bits.

Colour photographs suffer from decay, silver based black and white prints will hold out longer. Silver based negatives are more robust then colour, which is probably why studio archive on that.

Some filesystems, like ZFS, are designed to prevent bit-rot. You could try building or buying a FreeNAS box. And keep it running, letting it sit on a shelve is counterproductive. Might as well use it for editing and expand/replace disks as needed. Has some other neat features, like copy-on-write (and you know you're gonna love that).

But I never build such a thing so it's theorycrafting on my part.

Re: "Colour photographs suffer from decay,"

Not with UltraChrome pigment inks, these will last for 200+ years, providing you use the recommended media.

You are right about silver halide prints and negatives, these will have a far greater life, providing they have been fixed and washed correctly.

The inks and silver particles may last for years, but will the paper itself last this long?

I am sure with the growing popularity of digital data, manufacturers will turn their attention to archival matters.

BBC Radio 4 is transmitting a documentary this evening on "The Doomsday Book" data project that it undertook during the 1980s and how they managed to retrieve the data.

BBC - BBC Radio 4 Programmes - Archive on 4, Domesday Reloaded (http://www.bbc.co.uk/programmes/b0112913)

Shooting tapeless, and am using SD cards just like I used tapes - single use recording and archival. Their price is now on par with MiniDV digital masters in terms of hours stored per unit. It's easy to consolidate footage and toss bad takes, requiring fewer cards overall. They're tiny, and so easy to store anywhere. I have tiny tiny thumb drives from the early 2000's that still work, so I have some confidence that these archives will endure at least 10 years; then they can be transferred to whatever cheaper, better solution exists.

There is a big difference between an outdated file format and unreadable data. If a video file can be read, it can be converted to a format that can be read / played. But if the file is unreadable due to media corruption or drive failure, that's a whole different problem. I believe multiple copies on multiple devices / media types is a good way to minimize the risk of loss. Storing it in the 'cloud' is also a good idea but don't get too warm and fuzzy with that as the only archive. Hackers, sabotage, business failures etc. can spell the end to an online file as well. This isn't an issue of finding the best ONE way to archive a video file. If it's that important, store it in several different ways on different media and online archive as well.

Well Erik, there are a lot of differences between tape and disk. In the case of disk, the data is permanently associated with the disk drive. If you can't attach the drive to your system you can't read the data.

In the case of LTO at least, your cartridge is readable in a newer drive and you can be pretty confident of being able to retrieve the data for at least 20 years or more.

By the way, if you have a LOT of data, how about one of these??

Big Blue tapes up big data ? Channel Register (http://www.channelregister.co.uk/2011/05/09/ibm_ts1140_4tb_tape/)

Looks a bit out-of-budget, that IBM monster

I don't think you can expect to archive on any media, stuff it in a safe place and find it to be still readable ten, twenty years from now. Apart from having multiple copies, you'll likely have to refresh the archives regularly and connections, interfaces and protocols disappearing are not going to be an issue when you refresh your archives timely.

Refreshing an archive stored on harddisk costs about ten seconds of your time (never mind that your system will sweat on it for hours, or even days). I don't think any other medium can match that.

There is a big difference between an outdated file format and unreadable data. If a video file can be read, it can be converted to a format that can be read / played.

Add to that: As patents expire, ancient formats can be supported for free. Even if mpeg2 will eventually be replaced by mpeg4 or something completely different, I believe it will be supported simply because it doesn't cost extra.

In the case of LTO at least, your cartridge is readable in a newer drive and you can be pretty confident of being able to retrieve the data for at least 20 years or more.


I think it's not a good idea to plan for any media to last more than 5 years, 10 if you're lucky. Backward compatibility is a nice feature, but it also hold back advances in technology, I wouldn't expect it. In stead, I believe the continuous migration to newer media is a safer bet. Simply taking the past into the future. This works as long as storage capacity (as in GB/$) grows faster than my storage needs.

Say, this year I need 1TB and I'll get a 1TB drive, next year I need another 1TB, but as storage capacity grows exponentially, by that time, I'll get a 2TB drive for the cost of a 1TB drive today. Even if capacity doesn't double every year, it's good enough because it does grow exponentially while my needs grow linearly, as long as new formats doesn't enter the market, say as SD->HD or HD->4K. So, following this strategy, possibly I won't be able to read a five year old drive because of changes to the interfaces, but frankly I don't care, because the same data is on my 4, 3, 2 and 1 year old drives.

Just to compare HDD capacity against LTO: LTO has gone from 100GB in 2001 to 1.5TB in 2010, that's 15 times the initial, but HDD has a history of 100 fold capacity growth per decade. So, eventually tape just doesn't make sense in the GB/$ race even for long term archival, it becomes cheaper to copy data to newer disks than to invest in tapes. For what I can see, break even is just about now.

BR, Erik

I've heard around that we may actually start to see price increases in computer stuff in the next year or so. Partly due to higher labor cost in China and shortages of supply due to the Japan earthquake. Chinese labor cost has risen very significantly in the last year or two. Some Chinese companies are actually outsourcing manufacturing to Vietnam.

The typical analysis of total cost of tape vs disk assumes that the disks are running and sucking electricity and that tape cartridges are sitting quietly on a shelf or more often in a robotic library of some sort.

The reasoning is simple. Disks are engineered to spin, not to sit around for long periods of time. Tape cartridges are engineered to sit around. We have decades of experience in engineering both types of media. Some disk arrays support what is called MAID - ie massive Array of Inactive Disk - the arrays can power drives down either completely or to slow spin status depending on usage, This feature has been moderately well accepted in Japan but not in the US or Europe.

I don't know of any major disk user (except individuals or really small outfits) who store disk drives on the shelf. There is a technology called RDX (RDX Removable Disk Storage Solutions & Backup Technology | RDX (http://www.rdxstorage.com/)) that is trying to gain traction - it consists of drives with removable disk cartridges similar to tape cartridges,