Digital memories – how do you preserve them?
 

Not that I like to live on the bleeding edge of technology, but it sounds like a good idea to gather ALL my digital memories in one place. I got tired of scavenging through dusty boxes every time I hear the “honey, could we watch that Hawaii cruise again”?

Here’s what I am contemplating. My personal/family memories are at this point on VHS, Digi8, and mini DV tapes (both DV and HDV). I would capture all this stuff onto a hard drive, and organize it nicely into folders by year, month, etc. With hard drives getting dirt cheap, I can get 2TB worth of storage for just over $200, and configure it as RAID1 for data security. Once on the drive, I could edit my footage if needed, cut out bad takes, improve it, etc. I would also put all my still photos on this drive.

This would solve a major concern with family memories: how do we hand them down to the next generation. VHS tapes are fading, we need to save them as soon as possible, and digital tapes are following close; burned DVDs are just about as short-lived as well, so as of 2009 a hard drive seem to be the only long-term solution. If it gets filled up, in a few years I will be able to purchase a larger capacity for the same price or less, while the player keeps playing – and the next generation of players will be probably even better and cheaper.

For viewing, I would purchase a digital media player.

1. Is this whole idea good? What do YOU do for preserving your stuff?

2. What codec(s) would you use to store the video?

Thanks,

I've had this same exact question myself.

In my case, I copied my mini-DV collection to a hard drive. But its still hard keeping up with several 20+GB files that I rarely if ever access. Their baby videos, I don't want to losing the originals. I worry about keeping one or more hard drives around. They can be sensitive to static electricity, the interfaces change over time, ie PATA vs. SATA various speeds of PATA. Thinking about enclosures too, USB, vs FW400 vs, FW800, vs eSATA, etc.

My leaning has been to write it to optical disc. I personally haven't had an issue reading old media. I have seen my newer burners damage old rewriteable media because it can't write as slow as their maximum write speed.

I'm curious what people do with their video, especially videographers who do this for a living. Seems that you'd quickly end up with lots of big files that you don't want to erase, but you don't need to access quickly.

Would there be any interest in a program which to split up the files unto CD, DVD or even Blu-ray optical discs?

I was curious about this comment about DVDs being short lived.

If you mean physical decay, I googled and most results seem to indicate and optical media IS long lived. There are stories of them not lasting long but most seem attributed to excessive bending.
Optical media longevity
Optical media longevity

If the point is that DVDs are being replaced by Blu-ray or whatever else comes next, thats true of all electronic media.
CDs were introduced in 1982. Thats 27 years ago. And they are still being made and can still be read by most CD, DVD players and computers. Pretty impressive.
As I mentioned, hard drives have similar issues with interfaces, SCSI, PATA, SATA, USB1, USB2, FW400, FW800, eSATA.
And yet another point, the data that you write is in some compressed format. Finding a player which supports old formats will get harder over time.

Bottom line, there is no set it and forget it solution, but optical discs seem like a reasonably safe way to archive files for several years at least. In any case, make a copy of the set and store it offsite in case of fire or theft.

The best you can do is some form of tape. Current favorite is LTO. It's designed for both backup and archiving. Given the legal requirements for data archiving by many corporations in the US, whatever your favorite Fortune 500 company is using would be a good choice.

There are cheaper, and potentially less resilient formats are DVD, BD and USB hard drive. Stay away from CDs as they are too easily damaged. For now, my archiving choice is DVD since LTO is a little spendy for my needs right now. My guess is that DVDs will be around after I'm gone (and I'm pretty old) so then it becomes someone else's problem.

If you use hard drives, USB makes the most sense to me since nearly everything connects via that interface, but be prepared to take it down off the shelf and run scandisk against it every three to four months. Rotating magnetic doesn't take kindly to being ignored.

Firstly, I keep all the original tapes - Video8 (PAL and NTSC), Hi8 (PAL), DV (NTSC) and HDV (NTSC).

Video8 and Hi8 are dubbed to DV tape and it kept.

All the DV/HDV are dumped to my external hard drives (1 x 1TB holds all of them so far).

I create new files with the decent clips from the raw files. I keep the raw files.

I record the trimmed files on DV/HDV and keep those tapes. These will be the first port of call if I need to recapture the clips for any reason. I encode the trimmed versions to DVD which I use for primary playback in my standalone player (I rarely, if ever, watch anything for recreation on a computer).

I don't encode to any other format and only play the original tapes when absolutely necessary.

Can you say 'redundant'?!

I'd vote for optical (DVD). Hard drives are bound to fail even with a backup I just don't think they are reliable for long term storage. Myself I use RAID5 backup for my family photos, with an annual burn of everything to DVD. (photos, not video).

Video is on DVD also, at least the old analog stuff, and I have been waiting years for bluray to archive the digital tapes, which can stay as backups.

I'm not convinced anything is permanent, but I am hoping to preserve things so that by the time my kids have this task, they can take everything I did and add to it. By then I am assuming that it will be solid state format with maybe some new optical formats and my entire photo and video library will be on a single disk or thumb drive. but that's how it goes ..

I am still waiting to be able to convert my wife's parents film to digital format so that I can have everything together in one place. but for me, hard drives, even raid5, are only the daily / weekly backup and not long term.

I'm not tremendously optimistic about the longevity of new optical technologies.

The higher the density, the smaller the laser beam --> the less power can be delivered to the medium --> the need for a medium with less resistance to phase change --> the easier it is for the medium to revert to its original phase with time and temperature and humidity, etc.

Hard drives sitting on a shelf are also not great as archival media.

Of all the obvious choices, I think tape is about the best. Of course we then have the technical obsolescence issue, ie even if your tape is fine in 50 years, how will you find a drive to read it? The only real answer is to periodically copy from older technology media to newer. ditto for codecs, etc

Life was so much easier when people wrote on stone tablets!

On the other hand, even stone tablets got broken, worn down by the wind and sand and acid rain, and people forgot how to read them even if they survived intact.

Digital archiving is a heck of a challenge. Lots to think about.

Tape is prone to physical breakdown. The tape's substrate itself can start to degrade. An example is the space suits for the Mercury and Gemini astronauts: The synthetic materials they're made of is beginning to fall apart.

Hard drives are at risk of breaking down. Many moving parts, including a group of platters that spin at 7200 RPM.

I had a commercially pressed CD stored in an air conditioned room fail. And burned DVD's have failed.

It's impossible to say what formats or storage mediums will still be around in the long run. 8mm movies are tough to replay because no one is making projectors anymore. And, of course, everyone knows how hard it is to get a Beta deck.

I would suggest making multiple copies and distribute them to relatives. It's the same principle of fish laying a lot of eggs to ensure a few successful progeny. Perhaps store your copies on hard drives for fast and easy access, as well as rapid duplication. Make at least one backup to be physically stored somewhere away from the masters, just in case of fire, flood or other calamity. Use optical disk technology to make reasonably priced duplicates for distribution.

Try to include somewhere in there some plain text files describing what's on them, who is featured, dates, etc. Avoid using Word Document files because compatible software may or may not exist in the future. But plain old ASCII has always been decipherable since the 1970s.

For me, the easiest is a good hot swappable/trayless drive enclosure and a set of plastic hard drive cases. I find it's much more convenient than tape or optical disc.

MPEG2 or MP4/H.264?
 

For me optical disks are out of question! Yes, I use them all day, but I already had a few fail after after only a few years without any mechanical damage... and they ARE very easy to damage in various ways. Sure, I won't expose them to direct light, but that may still happen by accident, the dye can be easily destroyed. A friend of mine did some tests, some DVDs kept up for week, most failed within two-three days of exposure to direct sunlight - I know, I know, that's an extreme test.

Yes, I could *try* to take good care of them, and make a new copy every 3-5 years, but that's extremely time consuming when we talk 100 disks or more.

To me a hard drive based solution makes a lot more sense, making sure I have a reliable backup. Come to think, a two separate drives based solution sounds like more secure, keeping one drive in use, with a second drive in off-site storage (my local banks fireproof safe for example), both with identical content, syncing and swapping them from time to time, and adding content as needed. Tapes can go in the basement, for backup of the backup, I am sure they'll all fail in 15 years or less.

When do you guys expect for solid state drives to completely replace today's spinning drives? And when do you think they will become competitive in price?

Now, the big question is the format... do you foresee MP4/H.264 replace MPEG2?

And the bitrate... my own tests show that MPEG2 compressed with the best software compressor (for me, that's CinemaCraft), DV originated video is visually lossless even at 3-4Mbps (content dependent)... has anyone done any tests to see what this limit would be with H.264?

Thanks,

And how long would a hard drive last with the same test? Hard to imagine how you exposed it to days of direct sunlight unless it was outside in the weather.


Basement storage is risky because of mold. In general tape can break down. And its like a chain, all it takes is one weak spot on the tape to make it break or gum up the heads and you're in trouble.


Hasn't it already? I'm joking a little, but not much. MPEG2 is used for "legacy" technologies like DVD, but Blu-ray already supports H.264. There are lots of h.264 video cameras available.

Another question is how long H.264 will remain a leading contender. When will some new even better format come along and will licensing remain as onerous.

My friend taped the disks to the window glass, inside. Light is NOT a factor with hard drives, as the disk is enclosed in metal.

At the moment I keep everything in the original format... at least as long as I still have hard drive space. That means Windows DV for standard def (dvsd codec), and m2t for HDV.

When I make up my mind on the format, I'll compress it all... or maybe not, depends on how fast hard drive prices drop and disk space increases.

Right now all our show material is stored on mirrored RAIDs. I'm using Firmtek's hot-swap drive system with SoftRaid. This is probably one of the lower-cost high-capacity systems available and it's proven to be very reliable. Of course the drives have to be treated with kid gloves.

I have about 24 Hitachi SATA drives on the shelves, each holding between 500 to 1000 gigabytes. Hadn't had a failure in the past five years since I started using them. And it looks like SATA will be around for quite some time.

Solid state is a good idea. But I don't think anyone is certain about the longevity since there's not much discussion about using them for long-term storage. They're certainly rugged and seemingly reliable. And the price is coming down fairly fast.

Re - when do we expect solid state to completely replace spinning disks?

Probably never - or at least not for a very long time.

Being in the LTO magnetic tape business (albeit on the business side rather than technology, even though I was a techno-geek 50 years ago), I usually attend a once a year technology conference where representatives of all the major storage hardware and media cmpanies get together to assess the status of technology programs we fund at several universities. As part of the meeting, we usually produce a technology roadmap comparing the various storage technologies - everything from heat-assisted magnetic recording solid state, tape, optical, even holographic and yet more bleeding edge technologies like the "millipede" storage originally developed by IBM's Zurich research labs.

The net of it is that we don't see any earth-shaking changes in the cost/capacity/performance relationship of the various technologies - at least not in the next 5 to 10 year period.

Below are a couple of links to publicly viewable pages that give some interesting numbers for feasible storage capacities over the next several years. Almost all of the publications of this group are only available to member companies and research organizations so not much content here, just some big numbers.

http://www.insic.org/EHDR.pdf

http://www.insic.org/TAPE.pdf

And a couple of references to "millipede" storage

Millipede memory - Wikipedia, the free encyclopedia

http://www.nanochipinc.com/computer_world_pdf.pdf