STORAGE / WORKFLOW Help
 

Hey Guys,

I'm sure this has been discussed somewhere so if anyone could point me to the thread that would be great.

Using the EX-3 with FCP and doing a lot of corporate work. In the 'old' tape days, once finished a project we would delete all the raw files and just keep project files, masters etc and if need be, re-capture the raw at a later date. Now with file systems what should I be doing? To keep all the raw files of all jobs is insane in regards to space. Any ideas/suggestions. I'm new to the EX3 so is there anything that can be done as far as keeping offline, low res files that can be later reconnected to the original in full res?

Thanks in advance

Jason, welcome. Well, that is exactly what I am doing as are pretty much most of the EX owners. Some store on DL-DVD, others on the XDCAM disks that have just now been set up to write onto or, like myself, on hard drives. I store everything (raw BPAV folders & files) twice, on 2 separate drives. In fact, I have 2 Lacie 500Gb Ruggeds that I transfer the BPAV's to ALWAYS via Clip Browser with CRC ON. So straight away I have 2 separate copies. I then transfer the same files to the 2 hard drives in the editing computer. So, essentially, until I run out of Rugged space, I have the BPAV folder & files saved 4 times.

I have been doing this for almost 1 year now without any issue. Yep, get ready to store, store, store your raw files. It is the job that you trash that will need to be revisited one day in the future and if you haven't got the raw files.....

Where abouts are you based?

David is right. Just do be sure, even though you are on a Mac that you backup the entire BPAV folders using either clip browser or Shotput. Perhaps one suggestion is to backup the BPAV's onto one drive and the .movs on another. If you get a drive failure at least you do still have a copy of your clips. It's a good idea to keep the BPAV's as in the future it is likely that the need to use .mov's with FCP will go away :).

In terms of cost vs tape it's really not expensive. 500Gb usb drives are cheap and getting cheaper every day. With around 25 hours on a single drive your looking at $7 to $9 an hour for a pair of drives, that's less than pro-grade tape would cost. Plus you can keep one copy at the office and one somewhere else for added security.

Not wanting to hijack this post but I think I might not be doing this the correct way.

I copy my SDHC cards to a folder on my HD with Clip browser2.5 with crc on.

Then, this is were I think I'm going wrong, I just copy and paste that folder to another folder on an external HD. Is it OK to do it this way or should I use CB to copy the folder to the external HD too.

Thanks in advance.

Use clip browser always.

Paul.

Here is what I do:

I insert the express card into my laptop and use Shotput or Clip Browser to import them onto the laptop. If I have my USB/firewire external drive hooked up to the laptop, Shotput will transfer to that at the same time. If I don't have the external drive hooked up, I drag and drop the BPAV folders onto it later.

Then I take the external hard drive to my editing computer when I get the chance and hook it up via firewire. I drag and drop the BPAV folders onto one of the internal hard drives and use Clip Browser to create MXF files for my editor (Sony Vegas). Never have a problem. Not sure why the insistence that Clip Browser or Shotput always be used to transfer the BPAV folders anywhere they have to go. Once either of those programs have been used for the initial import from the SXS or Express Card Adapter they seem to remain intact no matter how you transfer them from one place to the other.

John

I use clip browser (with CRC on) to copy every time. It also ensures me that the first copy (using clip browser of course) worked properly and so on. Always (spot) check back the clips once they are copied.

Jason, there's a lot of good advice in the previous replies for archiving your raw footage, but I don't think that was really the point of your question. Was it?

When I finish a project I might have 80GB of raw footage, music files, graphics, animation files, etc. scattered all over the place on my drives. And maybe out of that 80GB of raw footage I only really used 20GB of it and the rest got left on the cutting room floor.

By using FCP's "Media Manager" function you can automatically make a copy of ONLY the stuff you used in the project, placed in one neat little folder, anywhere you want it. There's no recompressing and no harm done to the files. Media Manager even trims clips so you only archive what actually got used (for example, 10 seconds out of a two minute clip). You can even control how much handles are added to the trimmed clips.

This is a great way to condense a project down and archive only the stuff you really need. If you ever need to re-open the project in the future, it's all there, ready to go in a matter of seconds.

Obviously, this is in addition to archiving your BPAV folders. This is an archive of the "project" and not the raw footage. Don't forget to archive your raw footage someplace else.

Does that answer your original question?

When you guys say to use Clip Browser w/ CRC on, does that mean in Preferences you check the box that says ... Perform CRC check after copy?

B

(I just unboxed the camera this week and have yet to have time to start playing with clip browser and its realtionship to FCP, ect. So much to learn.)

Thanks for the advice all.

I second Doug's Media Manager tip. One of the best features of Final Cut, really good.

Also it is great for swapping projects between Macs.

Top DVD for the EX1 by the way Doug!

LTO-4 data-tape.

I'm one of those who believe data-tape is still the best form of long-term archiving.

Conceptually, I think you need to have three storage buckets: online storage for editing, near-line storage that is quickly accessible via Ethernet/eSATA/USB/Firewire and then long-term offline storage which is typically shelved media.

It's that last bit that I believe data-tape is best suited for and not hard-drives.

I posted this product line elsewhere but it's probably worth a re-link to:

Cache-A Products

I think we'll see more products like these as people begin to realize that shelving hard drives for archiving long term is not realistic after a certain point - either due to physical storage limitations or the reliability of a mechanical device that hasn't been started in years.

I think once you've properly integrated a tape-drive workflow into your routine, it scales perfectly and will be much more reliable in the long term.

Is it necessary to install the SxS driver that came on a CD when I bought my EX3?.

Yes, that's right.

oops. I was editing my post to fix a typo when my computer crashed and I lostmy original message.

I think this is about what I had written before:
I currently backup my projects on two hard drives. I will start backing up on 50GB optical discs as soon as I can get the firmware updated on my PDW-U1. The optical discs have proven to be 100% rock solid over the past 3.5 years that I've been using them.

Okay I thought it might be fun to share our workflow. Maybe it will help someone. We produce television commercials and corporate videos so 99% of our footage/projects must never be lost as someone has paid us to produce them.

Here's what I'm calling THE LAZY BUT CAUTIOUS WORKFLOW. It's been working really well for us since January of this year. We haven't lost a single piece of footage. (knocks on wood)

INGEST
1) Connect the SxS card to my MacBookPro
2) Connect a Western Digital 500GB buss powered FW800 external hard drive to MacBookPro
3) Launch XD CAM Clip Browser
4) Using XD CAM Clip Browser create a new folder on the WD 500GB external hard drive
5) Using XD CAM Clip Browser drag all clips from the SxS window into the WD 500GB hard drive window
6) If there is a second SxS card, insert it, select all the clips and drag them into the same folder. This is important because any clips that might have spanned the two cards will be automatically joined together into one clip.
NOTE: This is my BPAV archive drive. Once it fills up, I put it on the shelf and purchase a new one.

EDIT
1) Move the WD 500GB hard drive to our MacPro tower
2) Launch XDCAM Transfer
3) Click the ADD button and navigate to the new BPAV folder on the WD 500GB hard drive
4) In XD CAM Transfer name all the clips (sometimes just selecting all of them and giving them all the same name)
5) In XD CAM Transfer sort through the clips and label the good clips with OK
6) Change the Status View so that only clips labeled OK are shown
7) Go to XDCAM Transfer>Preferences>Import Tab and choose a location to save your MOV files to. We save them to our fiber channel RAID.
8) Select all the clips and click the IMPORT button

BACKUP
1) We have a FW800 drive that we use to back up our RAID. We use Apple Time Machine because it's easy, and we don't forget do back up on a regular basis.
2) Our RAID is striped RAID-5 so if one of the drives suddenly goes bad, the RAID switches to RAID-0 mode and you can continue editing until you replace the bad drive.

ARCHIVE
1) When the RAID gets full, we use EMC Retrospect Backup to archive our project files (MOV videos, graphics, audio, everything except the BPAV folders) off to Western Digital 500GB portable FW800 hard drives. Retrospect automatically flows the data from one drive to another as the fill up. But the best part of using this software for archiving is when it comes time to restore. You simply launch the program, find the data or project you want to restore, select RESTORE and the program tells you want drive(s) to install. Once you install the drive it automatically starts the restore process.
2) As discussed in the INGEST description, we save all our BPAV folders on to separate portable hard drives. Worst case if we were to loose all our project files, we would at least still have the original footage. Also, we'll sometimes re-import footage from the BPAV files for use in other projects.

Let me know if anyone finds this useful. (or is pissed off that I took the time to post it) :)

Looks like a great workflow to me, and it very close to what I do. Thanks for taking the time. Newbies should print it and post it on the wall.

Wow, thanks guys....what great feedback. I will have to sit down and go through all that but to it will definitely help.

David, We have a production company in Fitzroy and we do a lot of corporate work. I'm dealing with terabytes of footage over a dozen projects and that's when it is becoming difficult to get the workflow right. Will sit down over the wekend and digest all the comments.

Thanks for this. Is helpful.

Am surprised to hear you're using Western Digital drives. We've had several fail and have collegues who have had WD drives fail and - while searching the interwebs for info on reliable drives that are cheaper than the G-Techs I'm presently using - I came across nothing but loathing for the WD drives. Like really vehement hatred the world over for WD. I was surprised and surprised to hear that you've had good luck with them.

B

Thanks!! I'm glad you find it useful. (I'm flattered Doug....means a lot coming from you)

I don't have any good data on why we chose Western Digital. We used to use Lacie, but they had some bad problems with their power supplies. After about 4-5 of our drives going bad, we switched to WD. Honestly, the main reason is that Best Buy sells them and there's a Best Buy 2 blocks from our office. I'm crossing my fingers that they will hold up.

We will be upgrading our RAID soon. We're going to switch from fiber channel to eSATA. We're going to go with a 8TB G-Tech G-SPEED ES drive for our RAID and then a Drobo Pro for backup. I really like the expandability of the Drobo Pro. Hopefully that will make it more future proof than the drives we've purchased in the past, as you can add/remove drives for more storage capacity. This is something I wish you could do with a high-speed RAID. Why Drobo didn't release the Drobo Pro with an eSATA connection is beyond me. That's another reason we're waiting a bit before we purchase a new high-speed storage solution.

I have bought 6 Iomega drives and all have failed (!). Now I have more than 8 WD drives, including 3 passport, no one have failed (yet).

I have a dozen WD drives and no problems so far. Can't say the same for Lacie.

I have had mixed luck with Lacie's (older ones are better it seems), some have lasted 8 years some failed within 2 years, my first Lacie was a 20MB hooked to a Mac Plus, 1989 I think. I have more than 2 dz of those and about 25% have failed. The worst drives I have ever used are Maxtors.

Lately I have switched mostly to Graid (I have 8) and WD (I have 17) . No problems with either of those brands some are 4+ years old now.

These are just for client storage. Our working drives are all Raid 5 or 0.
For clients who pay for this service, we have always kept all digitized files and projects on FW drives so we can quickly restore old projects and re edit. Clients love this. I also encourage them to buy 2, one for us to store and one for them to keep.

The new file based is an absolutely wonderful addition in our studio, makes life so much easier, we also back all files up on Bluray/DVD and now have a database of all clips.

Lacie uses Western Digital drives (and perhaps other brands) inside their enclosures - they don't actually manufacture hard disks, just the enclosures. The problem with the Lacie's are their power supplies - they are notorious for failing. But if you replace the power supply, the drive and the data on them are fine. This has been our experience.

That has been my experience with them as well. Never actually lost any data - thought we did with the first failure until we figured out it was the power supply. All the other failures have been power supply failures, but when they failed to replace the last one due to it being out of warranty (even though it was a documented known problem), we switched to Western Digital.

Hi Guys,

Well I've been over everything and all the advice has been fantastic....great post Mitchell. However, Its not really solving the issue of space. I thought maybe that the BPAV folder would end up smaller than the FCP Mov's but its exactly the same. I guess we're just going to have to look at a really large RAID server. On average I would say we are capturing 2 hours of footage a week so this tends to add up when we're backing up our final projects to include all the renders, project files and raw files. Copying all this to external drives just doesn't cut it....I have 4 WD 2TB drives sitting full on my floor already, as well as 8 TB of RAID storage which is at its max. I think the only long term solution will be a SATA storage server with 2TB RAID drives. I think using FCP's Media Manager (Thanks Doug) is also a good option to save some space on certain projects so will give that a go too. Thanks everyone for a great discussion!

Cheers

I actually find HD/Bluray/DVD storage to be much easier and cheaper than tape.

I know some people shoot on HDSD cards and just archive those, now that is easy and about the same cost as tape. Also storage space is a lot better than hundreds of feet of temperature controlled tape storage. We don't do this, but I would consider it if I was shooting an incredible # of hrs.

We probably average 3-5 hrs of shooting here a week and I find that dealing with file based media is way easier than tape.

Thanks!

I'm reading between the lines of your post. It sounds like, like us, you're tired of buying an expensive high-speed RAID only to have to replace it after a year or two because it's full. The solution we're looking for is a RAID that would grow with you. Drobo almost has the solution with their Drobo Pro. When the drive fills up, just add more drives. When those fill up, replace some with bigger drives. But it has too big drawbacks....1) It's not fast enough. Why they don't release an eSATA version is beyond me. 2) It's limited to 16TB max. Actually I think this is a operating system limitation. But Apple's new Snow Leopard OS fixes that.

In the mean time we're looking at purchasing 2 x 8TB G-Tech G-SPEED es eSATA drives and a Drobo Pro for backup. It's a somewhat expensive solution, but still cheaper than a huge XSAN setup. We're still looking for other options over the next couple of months.

It's true that storage is really cheap these days, but high-speed storage is still expensive.

What are people typically doing "in the field" with their EX footage?

Are you making backups to your notebook and a portable hard drive? Two backup hard drives or one?

And do you use the Sony Clip Browser to shot log while out or during travel/downtime as well?

Finally, do you bring lots of SxS cards or rotate through just a few by offloading to a hard drive?

In the field:

I have 2 16GB and an 8GB SxS, a PHU-60 (Sony HD that mounts on EX3 and has 60GB almost 4 hrs rec time), and 4 16GB HDSD in M&R adapter.

I usually only use the SxS's. And offload at night to 2 Portable HD's hooked to my MBP, I use trans app to rewrap to .mov on one HD, and copy BPAV to the other.

If I need more than the 2 1/2 hrs worth of SxS, I use the PHU-60 first (4hrs). I am still reluctant to completely trust my SD cards. But I have used them for B-roll that is not crucial.

I plan to get 2 more 32GB SxS, they are best you can over crank any time and they are the fastest to transfer from. I have never had a problem with any of the media, even the SD cards.

If I had a shoot with minimal equipment, I would just have a bunch of SxS.

Similar to Olof's. I use two sxs 16's, two sxs 8's, and the phu-60, then two g-tech 500gb raid mini's to put identical copies of the BPAV folders on, dump, and then erase cards/phu.

It would be better to not have to dump during a shoot, as it's an accident waiting to happen, especially after a 12 hour day shooting, but I'm not yet trusting enuf of the mxr solution.

After shooting, dump the two field drives onto two identical 2 TB FW raid 0's (0+1) for editing, plus put original BPAV's on a raid 1 archive drive, plus backup to Blu-ray. Erase the field drives.

When the two edit drives get filled, I'll switch to a sata raid 5 unit.

Keep an eye out for this coming out this fall. Most people should be able to get by with having only two SxS cards in the field.

PXU-MS240 Mobile Storage Unit From Sony

You'll be able to off-load cards 10x faster than you can shoot them.

Hard Drives for Archiving?
 

I've been doing quite a lot of research recently on archiving.

For the cost and convenience hard drives are definitely the go, but there is a big issue of drives seizing up after sitting on a shelf for a couple of years. They seem to need to be spun regularly to keep the lubricants from solidifying.

Maybe the Drobo would help with this, but it will fill up eventually and you might have the same problem. Incidentally, I haven't figured out how the data is spread across the drives, and what the full drives actually contain.

I prefer solid state media for archiving, and I tried going down the Gold Archival DVD road (through ProDisc), but not only is it bloody expensive, it's also painfully slow and labour intensive (relatively speaking). Not a good trait for maintaining an archive system - pity because it seemed like the most stable option. Unfortunately, Blu Ray doesn't seem to much of an archival medium, as the pits are really small and therefore more error prone for storage.

Maybe Solid State Drives could be an option for "the future".

By the way, good idea about giving the clients a hard drive, that makes a lot of sense.

I've recently done quite a bit of research on the question of flash for "archiving" and the clear message I found was that flash memory is the worst possible choice for long-term or long-ish-term storage. Well, maybe it's better than rice paper.

Most of the conversations about flash focus on the longevity of the devices - i.e. 10000 erase-write cycles for MLC and 100000 cycles for SLC - but people don't often go into the retention of data on flash devices. The basic fundamental principle of how flash works is that each cell contains a floating gate that either holds a tiny charge or it doesn't. As that charge leaks away over time, the cell's contents become questionable.

The short of it is that retention is interactive with usage, so the more a device is written to, the shorter the retention period will be. This can sometimes end up being months and not years. And some consumer devices use really bad flash controller chips and allow "silent errors" to creep in undetected with the reasoning being that they're just consumer devices and just being used for silly photos and music and nobody will notice or care if a bit gets flipped now and again. To extend the retention of flash devices, you need to periodically (every few months or so) refresh/rewrite the data. This, of course, shortens the retention period of the device even further. Not a good way to go.

I know they're nice and small and easy to store and cheap and many here have a strong attachment to the idea of using consumer grade flash devices for long-term storage, but knowing what I now know, I won't trust any consumer MLC device to retain data beyond several weeks.

Hard drive and blu-ray/DVD for the projects inc footage and separate blu-rays just for the footage as a separate backup. Works for me. The raw footage also goes onto another HD which is stored on a separate site.

Make sure you spin the drives up from time to time though.

The mistake some people make is to assume archiving is "forever". Storage technology is moving so fast that in 5 years time they'll be a better, larger more reliable and cheaper solution. So just work on the basis that you are only storing the data on the medium term because you'll be moving it anyway in the future.

Media deterioration is an issue pretty much whatever digital format you use. I have premium Gold CD-R's from 10 years ago that won't play, but the cheap unbranded turquoise ones are just fine. Go figure! I would take all claims about media life with a pinch of salt. After all, if you look at most of the guarantees you'll find all you are entitled to is replacement media. Not much comfort if you've lost your only copy of a project.

I don't agree at all with this assessment. This is exactly what I do. Drives are so cheap now, that there is no reason not to keep all your full projects. I don't use RAIDs - don't trust them. They do nothing to protect against file system failures. Instead I use a mirrored drive. Basically I use a trayless SATA drive tower and have sets of drives, for example "Media 05" and "Media 05 Backup". I use Sync Backup to synchronize the drives nightly. When the drives are full I stick both on the shelf and get 2 new drives "Media 06" and "Media 06 Backup" I use Western Digital Green drives because they are fast enough and run a lot cooler (to my mind this makes them less likely to fail, but I don't have empirical evidence of that).

Additionally, when I'm done with a project a render out a ProRes self-contained Quicktime master, a version without music and titles, and individual audio tracks stored on an Archive drive with it's own mirrored copy.

There are a million ways to archive and backup. I've come up with a system that works great for me.

"I don't use RAIDs - don't trust them."

You do realize that mirrored drives = RAID 1?

I use the term "mirrored" loosely. It's not a real-time mirror, but rather a nightly synchronization. This eliminates the risk of a file system failure affecting both drives.

Ah ok. Gotcha. The terminology threw me.

As far as continuing the workflow into FCP, would you next:?

1) Open FCP
2) Import media (recently converted .mov's)
3) Set timeline settings to ProRes422
4) Drag clips to timeline and edit away

??