What's the big scare about copying BPAV?
 

Relatively new to the Ex-1r and the file workflow. Would like some feedback regarding copying, archiving and the somewhat confusing file designations.

Seems that some of us are casual about using the Mac OS finder or Windows to simply copy the BPAV folders from the memory cards to a hard drive. I've seen warnings from Sony and some editors that the initial copy process should be done with either Sony's clip browser software or an application like Shot Put pro. Shot Put has the ability to copy the cards to multiple locations simultaneously.
So far, I've separated (in my mind anyway), the archive and copy process from the editing. Once the archive is done, then there are multiple ways to import the clips into whatever NLE your using.
I see references to file types such as MP4, MXF etc.... Part of my confusion is how the clips get, or if and when they get transcoded. I thought the XDCAM format was mpeg 2 anyway.

I am very familiar with tape based workflow. I have a very uneasy feeling about this whole file workflow process as I feel lucky when the clips somehow manage to import in my NLE (final cut pro). However, I'm looking for a deeper understanding as to what's going on at the various stages of the process.
Anyone have a link to a better understanding? I've already purchased the Vortex DVD's.

thx

Just a few things.

The Sony camera places it's XDCamEX codec (which you rightly stated is Mpeg2 based) inside a file with an MP4 extention. That file structure does not have the ability to carry metadata. So the metadata is stored separately inside the BPAV along with an index and other information.

Doing an OS file copy is usually fine. But there is no verification on that process and if any small glitch happens, it can make the video file corrupt, it could show up in the video or audio, or you may see no effect at all. Programs like ClipBrowser do a verification process on the files that are copied to make certain they are bit for bit accurate and can be relied on.

Some NLEs cannot use the BPAV structure and so clipbrowser allows you to re-wrap the video/audio files into a .MOV file for use on the Mac, or a .MXF for use on the PC. Not all NLEs need this done. And if it is done, you lose the metadata. There is no transcode happening here.

I am unfamiliar with how Final Cut does what it does, but Sony Vegas can use either the BPAV structure or the MXF files. It does little with the metadata though. Avid on the other hand can use either the BPAV or the MXF fles, and if using the BPAV has access to ALL the metadata it seems. Very handy.

There really is a lot less to go wrong here than there is with tape. There is not verification process other than with your own two eyes when coming off tape. Tape also cannot carry metadata. Well at least not at our level. I can't even begin to tell you how nice it is to import clips into the NLE and sort by good takes automatically.

thx, explanation is pretty clear. How about renaming? Can you rename clips at any point? Or, should you rename after injest into the NLE? If you do that, it seems that going back to the original archive files with the renaming would make finding clips a small nightmare?

DO NOT RENAME THE CLIPS...

If you want to create a mess, this is about the fastest way to do it. Again, Metadata is your friend. I don't know what FCP gives you, but you should be able to do anything you need with the metadata and not touch the actual clip names.

This is actually one thing I love about Avid. You CAN rename the clips inside Avid to anything you like and Avid just doesn't care at all.

Just like Premiere... It's a great way to organize inside your project.

If you want you can rename the clips in FCP as well, it does not change the actual filenames. I tend not to do this but use comment fields for notes and folders/bins for organizing in FCP.

FCP is actually a nice database and quick search program as well as an editor, like I am sure Avid is as well.

I usually drop all the clips from a project into one FCP project and keep it in the same folder as the archived clips. It is much better IMHO than XDcamTrans for making notes and viewing clips, it will play back through my AJA Kona and Matrox MX02, so you can see what you really have.

Personally I do not keep my BPAV's once I have rewrapped to .movs. Because most of my projects are shot with a lot of different cameras and formats, so keeping everything as .movs is easiest for me. I use the nanoFlash a lot and it does not make BPAV's, it uses a higher bitrate EXcodec.

So personally I don't see much use for the BPAV file structure.

But before you have established your system of archiving and working, I would definitely keep the BPAV's. It also makes it easier to send clips to someone not using a Mac. Although that is easy by just trans coding the .movs in any number of different programs like Squeeze or Compressor.

I have personally never had a problem just drag copying BPAV's, over 800 hrs worth I think. But it probably is safer to use Clipbrowser or Shotput with verification. I always copy to two HD's and check a couple of the files by playing them back, before wiping the SxS cards.

For what it's worth...

We've been doing straight Windows OS file copies of BPAV folders from SxS cards to network archive storage since the PMW-EX1 came out and we have never had a corrupt file problem or anything similar; it's proven to be very reliable.

I've been told by a friend who is a a major computer geek that if you use the finder to drag & drop and check that the size of the file is exactly the same by checking with get info then you are safe. He says if the bits are the same it can't be corrupted.

I have had Clip Browser not see some files that were fine in the past but supposedly that's fixed tho it loeft me wary. I personally found the shot Put interface not simple enough and easy fro someone I gave the job to to screw up. Nice thing about Clip browser is that it will put many cards in one BPAV folder this connecting spanned cards. I always use it for spanned cards.

Someone else told me that Carbon Copy Cloner was also reliable because t double checked copies.

I use Toast to 'compare' the BPAV master to the two copies I make after I've done a drag and drop copy using the OS. It takes twice as long but ensures your copies are perfect. This is CRC checking.

Using any other system like Shotput or Clip Transfer will take just as long, the only way to make it faster is to use faster hard drives and a faster card reader.

Computers have error checking as a routine part of the file copy process. That goes for the Mac or PC.

If it weren't for that, you'd have all sorts of problems arising from making backups, installing programs, etc.

So you can use the clip browser for duplication and transfer, or you can just drag-and-drop, and save yourself a lot of additional effort.

Dean's right. There is no issue using drag and drop to copy files otherwise data would always be getting corrupted - it never does - as long as the files copy across completely the'll be fine.

Of course you must NEVER mess around with the file structure or rename files in the BPAV folder.

I also move BPAV folders around all the time with the Finder and have never had a problem. Clip Browser and XDCAM Transfer are important parts of may workflow, but that doesn't mean BPAV folders can't simply copied from one location to another without causing problems.

Ditto,

John

I have FCP and an EX1.

I use Calibrated Software's Quicktime Plugin "EX XDCAM MP4 Import" with native EX1 Files. This excellent software makes Quicktime think that the EX1 files are Quicktime and allows you to play them without transcoding anywhere Quicktime is used, including Final Cut Pro. I am really grateful that Calibrated created this plugin. It's saved me terabytes of uselessly wasted hard drive space and weeks of wrapping / transcoding time.

Finder Copies vs another tool:

Although the Finder or any OS based copy is quite reliable in normal copying, if one bit is off or corrupted in transit or if the source flash card destination hard drive is on the edge, you can get errors that are not reported in a Finder copy. It's also possible that a 'get info' would not report a difference in size. This is very rare but in fact I have experienced it a few times. Using a more fail safe method to copy, then compare the copy with the original, is more reliable. For EX1 I use the freely-supplied Sony EX Clip Browser software to perform the copies. It's easy, reliable, creates a destination directory with a unique name based on the date, and if you set the preferences correctly you can get what is called a CRC copy which ensures that the destination is the same as the original. You can then browse and playback the resultant copied EX video with it and rename the top level directory if you wish. Alternative methods, such as Shotput, or Chronosync with verification (which I use to transfer non EX flash media) is also possible, but not necessary because the Clip Browser is fast and reliable and has other browsing features.

Backing up entire directory vs other methods:

I would backup the entire flash drive to the hard drive, preserving the structure and meta data. Then you can always have the media in it's original form if you want to use it and other apps need that meta data to interpret the EX1 files properly, such as the FCP plugin or Calibrated's plugin. You can then import the files to FCP and perform the 'selects' there. I like to use 'subclips' to organize my FCP clips rather than renaming the clips. The file info is stored in the 'clip' and not altered but the subclip mechanism works well for me.

Alternatively, you can use Premiere CS5, which works natively with EX files, is faster than FCP. I am using that increasingly over FCP because it can natively work with pretty much every file type you throw at it.

Hope this helps.

I use Finder to copy my BPAV's all the time. Clip Browser is too slow for my purposes. However, I have had several instances where only the video files have copied and not the two metadata folders. Luckily I always check the files after copying and if I have time also open up the folder in XDCAM Transfer and check that they copied ok, so disaster was averted. I would recommend you always check the copied files before reformatting your cards.

On a SOMEWHAT related note, THANKS to everyone for this interesting discussion.

At my day gig, we JUST purchased an EX1r and I am the person responsible for the white paper on workflow. After reading this, I have implemented the Clip Browser copy method so that we have a verified file transfer for archive.

Works for me.

One thing the use of clip browser prevents is users delving in to the BPAV folders and only copying part of the required data. If you tell people that they must use Clip Browser then it can prevent all kinds of issues. Seasoned EX user know that you can use just about any tool to copy the data, but I get so many requests for help from people that have messed up their file structure by not copying the entire BPAV folder that I would classify it as a common problem. Tell them that they must use Clip Browser and the problem should not occur.

Here's my Apple FCP workflow. Been using it for 2 years and haven't had a problem yet. (crossing my fingers)

1) Format SxS card(s) in camera. (I never format/erase them using a computer)

2) Capture footage using EX3 camera

3) Copy data from SxS cards to a G-DRIVE archive drive using Clip Browser.
NOTE: Clip Browser both error checks the files and joins clips together that span more than one card

4) Using Final Cut Pro Log & Capture I import the files from the G-DRIVE hard drive to our G-SPEED RAID. This re-wraps them to MOV files.

5) We have a large Drobo Pro we use for backup. Using Apple Time Machine we automatically backup both the G-SPEED RAID (MOV files) and the G-DRIVE (BPAV files) to the Drobo Pro. This happens automatically many times throughout the day running in the background.

I've found this works too ......
 

Yes - an interesting thread.

My (in the field) method over the past nearly 2 years gives me peace of mind:

1) Copy BPAV folder straight from card to Hard drive (32Gb in 8mins via ExpressCard socket!)
2) Open up XDCAM Transfer and open the (disk based) BPAV in there. Check that all the clips display - I go through and randomly play a few.
3) If all ok, copy the BPAV to the 2 backup external drives - do the same quick check if time.
4) Ready to re-format SxS cards

I've found that any anomolies will mean that XDCAM Transfer gives all sorts of error messages, blank clips and you know something is wrong. I've only had problems once and that was using SDHC cards in an adapter. Decided to use SxS ever since without any problems.

I think that XDCAM Transfer is a bit overlooked. When I had the (above mentioned) problem, and a card got corrupted, I was able to navigate through to the MPEG files and it gave me nearly all my clips back. I was also able to salvage part of the clip which was being recorded when I got the error message. Not an experience that I ever want to have again though .......

Of course we should all be using XDCAM Browser now which is a combination of XDCAM Transfer Tool and EX Clip Browser in the same tool. Then in one application you can backup your BPAV's from your cards and export the .mov's that you need for the edit. No need to use log and transfer.

Think twice betore switching to XDCAM BROWSER
 

Well, I'd strongly disagree with that. I wouldn't advise any FCP user to switch to the new XDCAM Browser software. Version 1.0 is a step backwards for Sony and has created quite a stir even within the company. It's clear the software engineers in Japan just don't understand how things work in the real world. They took a great workflow and nearly derailed it. You're welcome to check my website if you want to hear my arguments against XDCAM Browser V1.0

XDCAM Browser 1.0 review:
Vortex Media: VIDEO & PHOTO Tools and Training


My advice is to stick with the traditional Clip Browser + XDCAM Transfer workflow. It works great. It's easy. And its very powerful.

I have not been using XDCAM Browser and just looking through it and Doug's comments I would agree with Doug that you need to think very carefully before making the switch.

Thx to all for laying out the various workflows. The thing that tee's me off is the lack of support anymore for the Express 34 slot on laptops. I happen to have an Apple laptop that has the slot but note that most new computers don't. (One exception is the Apple 17" laptop currently sold). Of course, this unit will cost you a boat load of dough.
My editing machine is a Mac Pro. Anyone have working solutions in play for this machine? I haven't seen any Express 34 slot cards.

Currently, using 2 methods, both awkward.

#1- copy cards via express 34 on laptop then out the firewire port to my archive drive (lacie ruggedized USB, Firewire, Sata portable drive). After copy, plug the Lacie drive into Mac Pro and copy files to the media drive. Time consuming.

#2- Use the camera's USB port to copy directly to the Mac Pro, using Shot Put Pro to make simultaneous copies to the Mac's media drive and the Lacie archive drive. Only issues here are speed and I don't want to wear out the port on the camera. It's bound to have mechanical problems sooner or later. I can't see that purchasing the 300 dollar Sony USB card reader is going to increase speed.

Note that I shoot long shows. Use 32 gig cards so copy time does play a role in my workflow.

All you need is a SBAC-US10. Best $270 you'll ever spend. Saves wear and tear on the camcorder and it's about twice as fast.

http://www.bhphotovideo.com/c/product/526430-REG/Sony_SBAC_US10_SBAC_US10_SxS_Memory_Card.html

Vortex Doug, ? about your website posting
 

Doug,
Read your review of XD browser, thanks for the info. I noticed the article below it talking about FC Pro SD workflow. What is the main reason you don't edit in HD and drop that sequence into an SD sequence at the end? Is it a render time issue or do you find a QUALITY difference in the end product?

thx
dave

The SBAC-US10 is faster than the camera, but not by that much IMHO as the performance is limited by the USB connection.

There are faster options but none of them come cheap. One is to use the Sony PXU-MS240 mobile storage unit connected via eSata. This can copy a full 32Gb card to a computer in about 7 minutes. Sonnet also have the QIO which is an express card (and others) to esata adapter specifically designed to work with SxS but I don't know of anyone using one of these.

Sonnet - Qio: Professional Universal Media Reader/Writer Plus Four eSATA Ports

I have one of the older style MacBook Pros with an express card slot and intend to hang on to it rather than upgrade.

David, there are several reasons why I don't edit with an HD timeline if I will not need any HD output.

1) My DVDs typically contain a lot of graphics, charts, animations, etc. and it is easier to create those at SD resolutions.

2) Video footage downscalses easily, but graphics and animations are trickier. By editing with an SD timeline I can see EXACTLY what they will look like, and thus avoid potential problems that wouldn't manifest themselves until after I'd rendered the whole thing. Some things that look good in HD don't look so great when downconverted.

3) I can scan & pan the HD footage to change the composition whenever I choose.

4) I can key-frame fake zooms or other camera moves.

5) There is no rendering at the end of the editing. Even a 2-hour program can be exported in just a few minutes. This is especially important if I get into the DVD authoring stage and notice any editing problems (such as a flash frame or graphics typo) that need to be corrected. If the sequence was done in HD, then I may have to re-render the whole thing and that could take hours. But, if the sequence is SD, I can make the change and export a new version in a matter of minutes.

6) I believe the quality of the finished DVD is superior if I have edited with an SD timeline rather than downconverting at the very end.

7) Other proprietary reasons that I choose not to disclose here.

Of course, if I do need HD output, then, of course, I will edit with an HD timeline and downcovert at the very end. Different workflows for different needs.

Here's a screen shot from an SD project I'm editing right now. Do I really want to create all this in HD just to throw away the HD timeline and downconvert at the very end? Seems like that would be a waste of time to me.

I have found the SBAC-US10 to be significantly faster than the camera. Yes, they are both USB, but the drive is still faster. I can't quote exact speeds because it has been so long since I used the camera, but, in my opinion, there's a big enough difference to make it worth it.

Doug,
thx for posting the extensive reply. I see your reasoning.