Archiving EX files
 

I just finished reading several threads on various methods for archiving the files from EX cameras.

Considering the suggestion made in another thread about cheaper cards, why not use the Kensington adapter with Extreme III cards for archiving? Nothing to break down (hard drives) and nothing to deteriorate (DVDs/Blu-rays)!

Just a thought.

It really depends on your budget.
16 GB can go on two DL-DVD for $3-$4. That's a lot less money than a 16GB SDHC. Then you have to make tiny labels too and some why to file them orderly.

I use surface printable DL-DVD so date of shoot and client name is on the surface of the disk. I put them in protective sleeve and in to my Vaultz file cabinet (holds 660 discs I believe). At some point I'll move to Blu-ray.

Craig...

your solution is fine, given that you don't factor the time you spend prepping and burning those DVD's. Bluray will be MUCH worse because of the time it takes to render and burn a Bluray format. Have you ever burned a bluray? Render times are on the order of 8-12 hours. I don't know what your hourly rates are, either for your time or the time on your machines, but, I'd be willing to bet it's not negligible. Perhaps after you factor this in, SDHC will look a little better. Or don't you value your own time? SDHC is available instantly, out of the camera, stores the native files...the good, the bad, and the ugly.

Bill, I believe Craig means burning data to a BDR to backup projects/footage. Not so much using "playable" BluRay discs as backup.

George/

Craig, you missed my point. Given the frailties of tapes, DVDs, Blu-ray discs, and hard drives what would be better, more secure in the long run than SDHC cards?

One 8GB card cost $80 (16GB $140). Amortize that and you're talking pennies a day. Seems a small price to pay for the most secure option available at this time. Besides, the cost is passed along to the client.

All things considered, how long will a SDHC card last compared to the other options?

I like using RAID storage on hard drives. Currently, I use two different ones. The first is this one - Thecus N2050 - Experience the speed of 3Gb/s on the world first eSATA on DAS | NAS server, Storage server expert

- which is just a housing. A pair of 1.5 TB drives in a RAID 1 and you have a redundant 1.5 TB drive solution for about $500. If a drive fails, just replace it. Data stays intact.

That's enough storage to hold the contents of nearly FIFTY 32GB SDHC cards. Plus, you can connect to your PC via the external SATA connector - making it very fast storage. Storing that much on 32BG cards would run $7000!

If you PC doesn't have external SATA, you can also use USB, but now your transfer rate will be a little slower.

Alternatively, there are several NAS solutions as well (network-attached-storage), but now you are limited to your network throughput. I have all GB networking, but the external SATA is still MUCH faster.

Without going into a diatribe here, a couple of things to consider:

The cost per GB as compared to SDLT or Blu-Ray DVD (Both data) is much higher.

And solid state memory is susceptible to any form of static electricity. As unlikely as this sounds, I do a lot of still photography and have had two cards scrambled while inserting them into the card reader. It didn't completely erase the disk but it scramble enough of the user bits that initially my computer wouldn't recognize it and then when we finally did get it to work I had lost about a third of the images.

For that reason the largest card I use in my still camera is 1GB.

Transcend 16Gb SDHC class 6 cards are available for $35, at a well known vendor. These are suitable for use in the kensington reader on an EX1. You can't convince me that an SDHC is any more or less susceptible to EMF than a hard drive. The SDHC has no moving parts and will last forever in the same kind of protected environment that you have to archive a hard drive, not to mention that it's quite a bit smaller.

I store every customer on their own drive/card. Each of my customers buys their archival media, so, they have a choice of SDHC or hard drive. I don't share drives between customers.
And for anyone who thinks that a RAID drive supported by the cost of NAS storage is a cheap solution, guess I'm wasting my breath. The non-recurring costs, alone, are a back breaker on a deal like that. The only time that's cost effective is for a business with an overhead that's probably bigger than all the capital in my business.

Chuck, I'm with you on that. Tape is still the best medium to archive to. Even though the LTO or SDLT drives are an investment, regular media is cheap and guaranteed for at least a decade. HP warrants LTO-4 Ultrium 1.6TB RW and WORM cartridges for up to 30 years archival life.

For corrupted data I have two words for you: Stellar Phoenix.

Bill, as to SDHC cards or flash memory in general; they've not proved themselves like tape has. How badly would your client freak out if you had sold them archiving only to find that data is corrupted after a few years.

Kingston states that "Flash storage devices are rated for up to 10 years under normal use", but "Important information should also be backed up on other media for long-term safekeeping". Normal use may be regularly writing data to the card (as it dynamically remaps failing sectors).

I however have no specific reliability/retention data on (new) SDHC cards, but IMHO when things get cheaper they don't usually get more reliable. I'm not saying that SDHC cards specifically are getting less reliable, just that I don't know.

George/

I think that now the SDHC cards are useable we can think about keeping the original cards. My clients pay an all in price for production services and it won't be long before the cost of cards can be absorbed in general charges.

That's my point, exactly, Bill!

The SDHC card appears to have all other media options beat insofar as shelf-life goes. Granted, the cost may be higher, but how much are your (client's) files worth. And talk about a time saver...!

I used to burn to DL-DVD. Because of the amount of time it takes to burn and the unreliableness I found with discs, I've switched to just backing up on two separate hard drives, putting them in anti-static bags and shelving them in two different places.

"Under normal use" is the key factor here. Having done some additional research, the life span of flash memory is dependent on the number of reads/writes. Using it as an archiving medium, meaning the read/write access would be at an ultra minimum at worst. Therefore, the life span of the card would be extended significantly!

Exactly right, Jay. The limitation to flash memory lifetime is the number of read/writes, and even that is about equal to a hard drive. As a storage medium, it's darn near perfect.

As far as your conservative approach, George, I wish you the best. My customers and I are quite happy with this solution. Personally, I despise tape. It's unreliable, it exhibits random dropouts, its bulky, heat sensitive, and sllllooooowwwww. And talk about sensitivity to magnetic fields...yikes! Maybe you should try super 16 film. You don't need storage media with it.

I have 6 computers here so burning to DL-DVD isn't too much of a problem. It's not fast though. It certainly doesn't prevent me from using the computer for other things either.

It's frustrating to have to remember to change discs every 30 minutes or so and type out a new label and then move the disc to a printer. It'll be a bit less of a bother with Blu-ray. I don't trust hard drives as back up. Even two of them. I can see hard drive and optical disc but not hard drive alone.

Again cost of SDHC is probably in line with using HDCAM but some clients aren't going to want to pay an additional $40/hr of media used. They SHOULD but many won't. Of course if they will then certainly that's a reasonable way to go.

So how are you labeling the SDHC cards and what are you storing them in (neat and orderly)?

SDHC can be stored in a fireproof vault. Commercially available vaults are available in a number of sources. The cards, themselves are tagged, manually, with a numbering system. The numbering system is managed, again manually, via a spreadsheet file. Not the best of systems, but, it works for a small business. Most of my customers keep their own archives. I don't maintain a library for them. So, my filing system is small enough for me that I can manage it, well enough.

In all fairness, I must say that finished edits don't go onto SDHC cards. Generally, the finished edit goes onto a DVD. However, the beauty of Avid is that the project files are small and the original media files are what's kept. Intermediate files are generally discarded because they can be re-imported from the originals at any time. The only exception are for FX renders, which have to be wrangled seperately.

But the SDHC cards are so small. What do you use to keep them from winding up in a big heap. There are file systems and cabinets for DVD, DV, DVCAM, many tape formats of course, hard drives can at least be stacked.

I don't know what's to prevent those cards from winding up in a big pile. Also, personally, I like being able to see client name and shoot date and reel (or disc) number on the media itself. I'd probably go nuts looking at 74869 (and what tiny numbers that would be!) and wondering what was on it and dashing off to the database to see. I guess my fantasy would be a tiny labeling printer and a storage unit that would fit the cards in slots.

I don't do this, but, why can't an SDHC project be stored in those small 3x5 manila envelopes. You can write whatever you wish on the outside of the envelope and store the envelope in a drawer rack built for such things.
Another possibility is to store multiple SDHC cards in those special glassine notebook pages made to store 35mm slides. Problems like inadvertently turning the notebook upside down and dumping out all the cards makes me shiver with fear, tho'.

Perhaps something like this would help.

(I'm sorry, I know I'm not being very helpful today, but I couldn't restrain myself.)

Ted, I got just the thing to help restrain you. It's a beautifully tailored white jacket. It comes with extra long sleeves and it ties in the back.

That should do it!

Craig, I would simply add that expense to the overall cost of the job. Play it up as an asset, not a liability! A measly $40/hr of media is nothing considering the client pays for the media anyway--tape for shooting and/or storage.

Worst case, shoot on your original SxS cards, copy those files to the SDHC cards for archiving--keep them for the client (charging an archiving fee) or hand them over to the client just like we did with tapes.

I honestly can't see it being done any simpler or cheaper.

ROFL... Thanks for the tip Bill.

Modern (data) tape systems are anything but slow, LTO4 goes up to 240MB/s 2:1 compressed (that is 840GB an hour) and is very reliable. They certainly are heat sensitive, but I doubt an SDHC card would hold up long on a hot plate. LTO uses such a very strong magnetic field so they are much pretty not sensitive to lower strength fields like those from monitors, magnets, security devices or any other "environmental" field.

LTO verifies the data off of the tape using a read head directly after the recording head; the recorded data is verified as it is written (and rewritten if verification fails). HP archival tape is warranted 30 years. A 800GB native (up to 1.6TB compressed, not video obviously) archival tape is under $80.

That is 25x 32GB cards worth of data (with a little compression and increased efficiency, probably 30 cards). Or $1750 worth of $35, 16GB cards for just $80. Also many projects would not fit on a single 16GB card, or even a 32GB.

I still use the Gen3 LTO with half the capacity at half the cost. A dedicated backup workstation with a mirrored disk staging area to put completed projects on before they go to tape archive. And I use the same system for daily backup and can easily take the data offsite for disaster recovery.

Restoring a 100GB project takes about 45 minutes, including retrieving the tape. I know you can't copy 7 16GB SHDC cards to disk that fast (or 4 BDR's or 12 DVD-DL). The newer tapedrives would be even faster.

I can start a backup/archiving action or a restore in 5 minutes and walk away. Even if it would take all day to finish, which it doesn't, it doesn't tie me down more than that. I'll take that over sitting there watching files copy over from a SD card for 15 minutes per card (at 20MB/s). Eject > Copy > Repeat (x7) - no thanks!

The book on this has already been written. And here's some pocket book/binder leaves for organizing SD cards.

George/

P.S. I have nothing against SDHC cards except that they are slow and unproven.

George...

your tape solution sounds pretty professional, aka hi-end. It also sounds pretty nice. Keep in mind that I haven't been in this business 35 years like a lot of you more seasoned guys. I have a very small operation, my overhead is zero, or I couldn't make this fly. I have fun, or I wouldn't do it. Wish I had started this business when I was much younger, but, oh well. Wishes don't count for much in this world. My training is in engineering(as are my degrees) I do know the technology, and perhaps that's a strength in that I know what new technology to put my faith in. I just don't have the history, know what I mean? And I envy you guys with history, you know how things got to where they are. Just don'tt be afraid of new stuff...some of it is pretty reliable...and pretty slick.

George,

Great link on the binder pages - thanks!

You may call it high-end, but really in the end it is just economical; remember, I am Dutch ;-) so if there is a better, "cheaper" way to do things I'm all for that.

I have an IT background, so the need for "backup" and archiving has been permanently imprinted on my brain. That's not always the case with you "video guys" :-)

You know the backup system here probably costs as much as an EX1 and I'm looking to upgrade to the LTO4 system early next year, which costs are probably closer to that of an EX3 (US pricing). Ouch. So if there's something that beats it, bring it on.

But projects are getting bigger with every camera upgrade, more powerful systems and more data is continually kept on online storage. A good backup and archival regimen is a necessity for me.

I can't afford a freak cascading fault to lead to a probable catastrophic business failure. And those are the most common; when you "forget" to do something (like checking backup logs) or do something that you really shouldn't - like putting projects on scratch disks that aren't backed up as you don't have room elsewhere - and then something else fails (a disk, human error, etc.) and "y're screwed".

So even though we've only got a handful of systems, this was necessary, not luxury. But I know this won't be for everyone. It just depends on the type of work you do and if you're a freelance shooter or mainly do events, this probably will not suit you at all.

And as long as you are (relatively) happy with your chosen solution and don't feel like it isn't up to the job or wasting your time, you're good.

George/

P.S. Thanks Ted. Thought maybe number the cards and put a paper card in the pouch with that number + an index or such.

Here's an interesting article that carries with it some promising news!

http://www.sandisk.com/Corporate/Pre...e.aspx?ID=4353

Hi Jay, I saw that around when it came out in July. Just read this: "SanDisk SD WORM cards are available now worldwide in 128-megabyte capacity and are expected to be available in higher capacities later in the year. Pricing is available on request." (emphasis added).

That's not going to solve our archiving issues just yet. From what I recall it was intended for medical info or such (maybe legal?) and had been announced years earlier. Aparently there are some technological issues there...

George/

Re-read my post, George. I said "promising news," as in the near future!

Sorry, I didn't mean to waste your breath.

I'm not sure what RAID drives and NAS have to do with tape back up?

I have five FCP edit bays and need to back up about 4TB's per month. WHILE Shooting to tape the original tape is an integral part of our current archival strategy. Now that we are switching to the EX3 I am looking for more efficient, secure and easy to integrate into our post production environment archival strategy. Also in all likely hood we will add two or three more edit bays resulting in an additional 3TB's per month.

It seems to me, unless I'm missing something, that the cost per gig for DLT is much lower than solid state memory. Don't get me wrong, if what your suggesting worked better and was cheaper I'd be very glad to use it.

Chuck...
I'm sorry if I sounded rude. It's just that backup and archiving processes are pretty different for a business that has massive amounts of data to backup and retrieve, than for a small, home owned business like mine. NAS is something that requires an investment in hardware/software infrastructure that can't be justified for a small business. It's just not cost effective. The cost of installing such a system for my business would never pay back the investment.

One of my complaints with this otherwise excellent forum is that there are many different types of users who read and post here. Clearly, I think, not all architectures scale with the size of the business. I would never presume to advise someone on your scale of business with my processes. I would think the reverse would likewise be true. I think it's obvious that manual storage and retrieval of even 32Gb flash cards would be cumbersome and inefficient for your needs.

I have observed that people jump into discussions with advice, criticisms, judgments, arguments when their input is not necessarily relevant to the discussion. Such is the case, IMHO for NAS RAID backup systems and onesy/twosy backup systems.

Hey, Bill -

I have a very small business, and prefer the NAS or RAID system. My little RAID is 500GB is cost about $400. However, going with larger drives (1.5TB) does not cost much more yet provides many more GB per dollar!

Another solution that I've used for years that I really like, is a SATA drive bay in my editing system. A SATA card ($50 to $75) and a SATA bay ($30) and you can hot swap any SATA hard drive, fill it up, and place it on a shelf. For critical data, I will write it to two different drives. Drives only run about $130 for 1TB - so it is extremely cheap as well. I know many people don't like hard drives for storage, but I only have a need to retain data for about five years. Since the drives are barely used, stored air conditioned space, and not subject to abuse, they seem to do well. I'm only on year 3 of this system, but it has worked well.

The best part is when I need to refer to or load up some old media. I slide the drive in my media bay, Windows recognizes it, and the content is directly editable and I can load and run projects right from it at SATA II speeds.

hmmm....
yeah, I've got 8, count 'em, 8 SATA II ports for various asundry external drives each on two machines.. Each drive holds its own project, sometimes work on 3 projects in a day, SATA external bluray burner that moves between 3 workstations. Some external SATA's move between my PC's and a MAC workstation. Use MACDRIVE so I can read HFS+ drives off of those Macs.

That's all well and good but I hate taking up space on an external drive with native files. I like to transcode them to intermediates/proxies, then archive the originals. Best option for archiving the originals seems to be CF cards. I have a slew of old IDE drives for storage, but, theyre very s-l-o-w.

So, for any reasonably efficient NAS you need a 10Gbit switch, router and server, plus, fibre optic connections would be nice to not bog the system down, Nothing beats the speed of moving a project drive between machines ILO trying to feed a 30 Gb file over a network. Then you need RAID cards plus RAID enclosures, and make that RAID1...RAID0 is too dangerous for project files.

Do you get where I'm going with this? All very nice, but, $$$$$$$$$

Archiving - for what purpose?

To just store the client data, why not use a large RAID1 (very reliable, even if the controller card goes down, you still have 2 identical copies.)

If you are talking data security as in "off-site" storage, then I guess rotating DLT tape backups would be the ticket - but I personally have never done it.

It looks to me that the chance of catastrophic data loss is remote if you have a redundant storage system (RAID1).

Seems like other cases involving theft/fire/equipment failure should be covered by business insurance?

Drobo
 

I have used a myriad of solutions for archiving. The problem is nowadays we have massive amounts of data to back up, it makes archiving on optical or data tape impractical (cause it's too slow and inaccessible).

Backing up to a hard disk is really the only current practical solution to the current need to archive massive amounts of video. I have removable caddy systems for IDE and now SATA, but I've felt that unless there is a way to monitor the drive for it's 'health' there really is no guarantee that the drive will lose data or become inoperable while on the shelf. It's just a mechanical device that can either demagnetize or just break.

Raid 1 is simple and fine, if one drive goes down you replace it without data loss.

Recently a company called Data Robotics has introduced something called a Drobo. It's basically a unit that accepts up to 4 SATA drives and connects to your computer via USB 2 or Firewire. I basically uses a RAID like scheme that allows 1 of the 4 drives to go bad without losing any data. If one of the drives does go bad, you can replace that drive and it will go on functioning. The capacity is basically equivalent to 3 of the 4 drives (ie 4 TB drives = 3 TB of storage capacity).

I like the unit, it's not cheap but if you add up the cost for similar solutions, like RAID 1, or RAID 5, or RAID 0+1, it's pretty competitive. The also have add-ons to turn the device into a NAS. (I have it hooked in via Firewire 800 into a NAS like solution of my own, an older Macbook Pro on Gigabit ethernet.)

It's also really simple to use, the software is smart and it's fairly compact and quiet. I have one now and I'm probably going to standardize on them in the future (I hope they stay in business.)

Personally, I prefer the safety in numbers option. Drobo looks like a good solution giving further protection as an external drive (array), but I would be worried about trusting it with the only copy of my files, even if it is spanning them across multiple disks.

I've worked as an IT admin in the past and have witnessed many nasty incidents of RAID devices losing all data. They do protect from hard drive failure, but if they fail themselves the results can be just as destructive.

Is it possible to pull a single drive from the drobo as an archive that could be used inside a computer at a later date, or is the data stored in a way which would be impossible to read without the use of another drobo?

I use a dedicated internal drive for storing raw EX data (BPAV's) and then use time machine on the mac to back up all of the files to an external USB drive, once it's full it goes into storage and is replaced by a bigger external drive that backs up the same files but gives room for more. This seems to be working out pretty well at the moment as the drives are getting bigger and cheaper all the time.

I like knowing that I have archives of my older files across multiple inactive drives and that my newer files are on 2 active drives all the time.

Time machine also has the added benefit of being able to cycle back through various versions of project files as they are being worked on.

I don't think you can, Paul. Just asked a guy who owns one that same question and he said you can't pull a drive out of the drobo for separate use, as it's spanning the info over multiple drives. When your drive(s) is full, you have to put in another, larger one.

I'm thinking that for archiving, I'll try a hybrid system: backup onto blu-ray discs plus a single hard drive that gets put on a shelf. Don't see the value of having 4 discs spinning a la drobo or raid, if the purpose is to archive dailies.

......unless there is a way to monitor the drive for it's 'health' there really is no guarantee that the drive will lose data or become inoperable while on the shelf. It's just a mechanical device that can either demagnetize or just break.
===================

I have never heard of that. If the drive is used only for archiving data it will be far far below the MTBF for the drive. The possibility of drive failure while on the shelf seems extremely remote.

John

Mike's correct.

When data is striped across multiple drives, they are a married set and must remain together. In your scenario, which is probably a RAID 5, fault tolerance is in effect due to shared information. Once one drive is removed, the remaining three still contain 100% of the data. However, you cannot read that drive by itself or move it into another existing RAID 5 and expect it to retain it's data. The benefit to RAID 5 over other RAIDs is that only one drive of the set is used for redundancy. If you used ten 100GB drives, you'd have 900GB of storage

However, with a RAID 1, a simple mirror, both drives contain the same data. They are written to and read from as a unit. When one fails, like with a RAID 5, you are left with all of your data intact. The benefit is that a drive can be removed and read by itself. The downside, you only see half of your actual storage capacity. In my opinon, considering all the things that can and do go wrong, a RAID 1 is slightly more fault tolerant.

John,

This has been my experience as well. Some of my drives, though small, are just shy of ten years old. Also, though I've had five or six fail over the years, I've never had one go bad on the shelf (and I usually have at least eight or ten on the shelf).

Exactly.

That is why I never bothered with Raid 1 setups for improved speed over the years. It wasn't worth risking data loss in the event of one of the two drives failing.

John