Best lossless codec for archiving? at DVinfo.net

I J Walton · August 15th, 2008, 01:33 PM

Hello there. I am about to archive my short film collection onto some external HDD's. At the moment they are uncompressed and very large files. This isn't too bad, but I would prefer to make them slightly smaller using a small ammount of compression.

I know of Huffyuv and Lagarith but I am worried about the future-proofing of them. If I encoded in huffyuv I assume that I could not read the files on another computer that does not have the codec installed.

My question is, other than uncompressed, what are the other best options for archiving to HDD? (As future-proof and compatible as possible).

Thanks.

Graham Hickling · August 15th, 2008, 05:43 PM

Well for any of the "lossless" codecs, you will face the need to have that codec on the computer you are using in later years. So since huffyuv is freeware and tiny, I'd actually consider that quite a good option.

The bigger challenge may be to guess which codecs will still be easily installable on future operating systems?

James Brill · August 15th, 2008, 06:09 PM

In all honesty I think tapes would be the best thing. Get some long run evals of HDcam or Digibeta and just lay them off.

Brian Drysdale · August 16th, 2008, 03:32 AM

Quote:

Originally Posted by James Brill

In all honesty I think tapes would be the best thing. Get some long run evals of HDcam or Digibeta and just lay them off.

The industry standard for data backup seems to be settling on using LTO, as used by IT people in banks etc.

Linear Tape-Open - Wikipedia, the free encyclopedia

Mark Keck · August 16th, 2008, 05:10 AM

I place all project files into a folder and zip it... Usually get ~20-30% compression for the whole project; obviously some files more than others. But it is completely lossless, and I don't have to worry about future proofing the codec other than choosing the right one to work in to begin with. Since you're uncompressed that shouldn't be a problem and I would think you'll get a fair amount of compression.

I J Walton · August 16th, 2008, 08:21 AM

Thanks for the replies guys.

Mark, I never thought of ziping an avi file for archive. It seems like a good idea so I will give it a try. Thanks.

Greg Laves · August 16th, 2008, 08:30 AM

Quote:

Originally Posted by James Brill

In all honesty I think tapes would be the best thing. Get some long run evals of HDcam or Digibeta and just lay them off.

I haven't done any recent research into the subject but tape manufacturers used to say that tape stock has a life expectancy of 10 to 12 years. I have succesfully played back tapes that were much older but I always wondered which tape would have the magnetic material come unglued and take down whatever piece of gear I was playing it in. Sort of like playing russian roulette. Plus, more care must be given to proper storage of tape stock.

Brian Drysdale · August 16th, 2008, 09:55 AM

Quote:

Originally Posted by Greg Laves

I haven't done any recent research into the subject but tape manufacturers used to say that tape stock has a life expectancy of 10 to 12 years. I have succesfully played back tapes that were much older but I always wondered which tape would have the magnetic material come unglued and take down whatever piece of gear I was playing it in. Sort of like playing russian roulette. Plus, more care must be given to proper storage of tape stock.

Unfortunately, hard drives and DVDs appear to be be worse or at least unproven, for long term archiving B & W film is currently the best.

Jim Andrada · August 17th, 2008, 12:19 AM

The next generation of LTO drives, due out the end of next year (approximately) will store 1.6 terabytes of uncompressed data per cartridge. The generation after that in roughly another two years will store 3.2 terabytes per cartridge. There are at least 2 to 3 million LTO tape drives in the field. IBM passed the million mark last April and we continue to see projections of 400k to 500k drives per year total across all manufacturers.

With this many drives in the market, LTO will be here for a while yet.

I was at a technical storage industry symposium last week in San Diego and a gentleman from Sony gave a presentation on how they were going to get to a terabyte per disc with Blu-Ray. They don't see getting past 400 to 500 gigabytes with a system that will read today's discs. There doesn't seem to be any real chance of using a shorter wavelength laser but they think they can get up to 8 layers on a disc. Tricky stuff - they will need variying thicknesses of spacer material between the layers to prevent unwanted optical interference.

Data rate will never match what LTO can do.

There are a few technologies that will eventually be capable of 10 terabits per square inch, both on HDD and SDD. It's doubtful that tape will get to these bit densities due to the need to read and write on different drives and interchange cartridges, as well as the reality of dealing with flexible media. Tape thickness will be going to around 5 microns and length per cartridge is already approaching about a half mile.

Ferit Yazgan · August 28th, 2008, 02:46 AM

hey guys, ok, tape backup is the best.. but it seems, buying a 1TB hdd and archieving is the best solution for me (and sure for other amateurs/home users)..

well, back to question, .m2t files are huge files, how can and which format can I compress them into?

anyone could give the instructions and links for compression huge .m2t files to store on HDD??
thank you.

------------------
Sony FX1E

Graham Hickling · August 28th, 2008, 06:01 AM

There is no "lossless" codec that you can use that will produce files smaller than your raw m2t files from the camera - those are mpeg2, which is a highly compressed format.

A good solution currently is Cineform, which uses wavelet compression to produce highquality avi or mov files that aren't too much larger than the source m2ts. The encoder is costly, but the decoder is free and can be installed on an unlimited number of machines.

August 15th, 2008, 01:33 PM	#1
I J Walton Regular Crew Join Date: Aug 2005 Posts: 40	Best lossless codec for archiving? Hello there. I am about to archive my short film collection onto some external HDD's. At the moment they are uncompressed and very large files. This isn't too bad, but I would prefer to make them slightly smaller using a small ammount of compression. I know of Huffyuv and Lagarith but I am worried about the future-proofing of them. If I encoded in huffyuv I assume that I could not read the files on another computer that does not have the codec installed. My question is, other than uncompressed, what are the other best options for archiving to HDD? (As future-proof and compatible as possible). Thanks.

August 28th, 2008, 02:46 AM	#10
Ferit Yazgan Tourist Join Date: Aug 2008 Location: Istanbul Posts: 2	then, which lossless codec??? hey guys, ok, tape backup is the best.. but it seems, buying a 1TB hdd and archieving is the best solution for me (and sure for other amateurs/home users).. well, back to question, .m2t files are huge files, how can and which format can I compress them into? anyone could give the instructions and links for compression huge .m2t files to store on HDD?? thank you. ------------------ Sony FX1E

August 15th, 2008, 05:43 PM	#2
Graham Hickling Trustee Join Date: May 2004 Location: Knoxville, Tennessee Posts: 1,669	Well for any of the "lossless" codecs, you will face the need to have that codec on the computer you are using in later years. So since huffyuv is freeware and tiny, I'd actually consider that quite a good option. The bigger challenge may be to guess which codecs will still be easily installable on future operating systems?

August 15th, 2008, 06:09 PM	#3
James Brill Major Player Join Date: Aug 2007 Location: Goleta, CA Posts: 299	In all honesty I think tapes would be the best thing. Get some long run evals of HDcam or Digibeta and just lay them off.

August 16th, 2008, 05:10 AM	#5
Mark Keck Regular Crew Join Date: May 2007 Location: Penang, Malaysia Posts: 123	I place all project files into a folder and zip it... Usually get ~20-30% compression for the whole project; obviously some files more than others. But it is completely lossless, and I don't have to worry about future proofing the codec other than choosing the right one to work in to begin with. Since you're uncompressed that shouldn't be a problem and I would think you'll get a fair amount of compression.

August 16th, 2008, 08:21 AM	#6
I J Walton Regular Crew Join Date: Aug 2005 Posts: 40	Thanks for the replies guys. Mark, I never thought of ziping an avi file for archive. It seems like a good idea so I will give it a try. Thanks.

August 17th, 2008, 12:19 AM	#9
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	The next generation of LTO drives, due out the end of next year (approximately) will store 1.6 terabytes of uncompressed data per cartridge. The generation after that in roughly another two years will store 3.2 terabytes per cartridge. There are at least 2 to 3 million LTO tape drives in the field. IBM passed the million mark last April and we continue to see projections of 400k to 500k drives per year total across all manufacturers. With this many drives in the market, LTO will be here for a while yet. I was at a technical storage industry symposium last week in San Diego and a gentleman from Sony gave a presentation on how they were going to get to a terabyte per disc with Blu-Ray. They don't see getting past 400 to 500 gigabytes with a system that will read today's discs. There doesn't seem to be any real chance of using a shorter wavelength laser but they think they can get up to 8 layers on a disc. Tricky stuff - they will need variying thicknesses of spacer material between the layers to prevent unwanted optical interference. Data rate will never match what LTO can do. There are a few technologies that will eventually be capable of 10 terabits per square inch, both on HDD and SDD. It's doubtful that tape will get to these bit densities due to the need to read and write on different drives and interchange cartridges, as well as the reality of dealing with flexible media. Tape thickness will be going to around 5 microns and length per cartridge is already approaching about a half mile.

August 28th, 2008, 06:01 AM	#11
Graham Hickling Trustee Join Date: May 2004 Location: Knoxville, Tennessee Posts: 1,669	There is no "lossless" codec that you can use that will produce files smaller than your raw m2t files from the camera - those are mpeg2, which is a highly compressed format. A good solution currently is Cineform, which uses wavelet compression to produce highquality avi or mov files that aren't too much larger than the source m2ts. The encoder is costly, but the decoder is free and can be installed on an unlimited number of machines.