Clip Browser failing - and not letting me know?!
 

Hello everybody, I just had a major problem with Clip Browser, latest version (2.6).

I downloaded 2GB of video on a SanDisk Ultra II Class 2 card connected to the computer through an USB Card Reader - never had any kind of problems.

Priority: data protection.
Perform CRC check.

So everything should be perfect, right? But when I checked the downloaded files, some of them were corrupt. Some wouldn't open, some would stop playing at one point, one even had interference with a clip I downloaded a long time ago and had deleted already! But the worst part is... some would play perfectly, and Clip Browser never told me there was a problem. I tried downloading them twice. Both times had problems in different clips.

We could think it's a problem with my hard drive, but, I copied the files just using windows explorer and there was no problem at all. Why would a very long and supposedely secure copy by Clip Browser fail, while a simple drag and drop in explorer was lighting fast in comparison - with no errors?

Is there another copy-paste program with security checks we should/could be using? I'm sure there are plenty, made for example to safely make backups or move data.

Thanks for your insight,

Ivan.

Personally I copy using the OSX browser and use the 'compare' function in Roxio Toast.

We've always just used a simple folder drag-and-drop in Windows and have never had an issue after about 4 terabytes worth of data transfers from SxS card to our RAID-6 Archive.

Why blame Clip Browser?

It seems to be you are placing blame where it doesn't belong. You either use real SxS cards or you gamble with your business and reputation. Was it worth it?

Okay, at the risk of sounding like a a**hole, I'm going to say I told you so. I'm really tired of reading about instance after instance of people being shocked that SDHC cards are not as reliable as SxS. NO KIDDING!

Then I get personal emails almost daily from potential XDCAM customers who are put off by all the horror stories they hear and expect me to reassure them they are going to make the right decision. Well, you guys that are cutting corners and trying to save a buck are making it harder for the rest of us to convince clients, friends, and associates that XDCAM is safe and secure. Those of you who use anything but real SxS cards, and then complain publicly when you have problems, hurt all of us. I'm tired of it.

If you want to use SDHC cards and suffer any problems quietly, that's fine. But when you choose to go public, then you are hurting me and every other professional else who has invested the money to play it safe and avoid the false economy of cheap cards.

I'm sorry to hear about any situation where someone loses footage, but if you play in the road long enough, you're going to get run over. And I'm not going to shed a tear. You shouldn't have been playing in the road at all when there is a perfectly safe alternative.

I realize this post seems mean, and is probably not going to win any friends, but I think it needs to be said.

Doug - I agree with you, to a point. But I think what is in question here is the reliability of the Sony Clip Browser to identify bad media and/or corrupt files.

No question about it - if you are using cheap media, you are gambling - period.

However, in this case the Sony Clip Browser software - configured with the data protection CRC check - essentially gave this media a "stamp of approval" during the transfer from the cards and obviously it was in error when it did this as the media was clearly corrupt.

This brings into question how much confidence we should have in the CRC check that is being done. It seems using cheap media - where file corruption is more possible - would logically suggest that media would be more likely to fail a CRC check. But that's not what has happened here.

Therefore, can I really trust the Sony Clip Browser CRC check during transfers from my genuine Sony high-priced SxS cards when it's also giving the thumbs up to corrupted files coming off cheap media?

Correct me if I am wrong, but the purpose of the CRC check is to make sure the copied data and the original data are exactly the same. It does not guarantee that the original was any good in the first place. How could the CRC know that unless it took the time to actually attempt to playback every single frame on the card? Do you really want to wait 60 minutes to copy a 16GB card?

If you've got bad clips on the card, CRC will make sure that those problems gets copied perfectly over to the hard drive. That's all CRC is going to do for you. CRC does not give your video a "stamp of appoval", it gives the copying of that data a stamp of approval. That's an important difference to understand. It sounds like CRC worked flawlessly if you ask me. If the data on the card is oka, then another attempt to copy should work, but it didn't. If he video playback on he camea is okay, then maybe a live capture via HD-SDI is a good solution to salvage the footage. But if the camera can't even playback the footage, then we know exactly where the problem lies . . . a cheap card.

I stand by my statement that it is wrong to blame Clip Browser, when the real culprit is sub-standard, non-SxS, non-Sony media. You get what you pay for, and if you're using the camera for professional reasons, this isn't a time cut corners. Look, even if I bought several 32 GB cards per year, I'd still be spending a fraction of what I used to spend on tape. Fortunately, I haven't even spent a dime on additional cards in two years. Now that's savings I can believe in. Using cheap cards is gambling with your livelihood. If you can't afford real SxS cards, you can't afford to use an XDCAM EX camcorder. Same thing if you buy a luxury car that requires premium gas and then try to put regular unleaded in it. If you can't afford the fuel, you can't afford the car.

Class 2?
 

Pardon me if I missed something here, but you were trying to write video to a Class 2 card? I thought you had to have no less than a Class 6 and even that has proven to be occasionally problematic. I'm with Doug on this one.

Class 2 cards are OK for stills but not for video work, I am surprised you even managed to record on them without getting a media error message.

I agree with Doug also. I do use SDHC without any problems but have in the back of my mind the thought that SxS cards are what Sony guarantee for use. When postings come up that ask if some odd make of card is useable or has given problems I despair. One good thing to come out of this is Doug's reminder that CRC is only confirming that files have been transferred not that they are necessarily good.

Exactly. Which is why I found Doug's rant baffling. SxS does go wrong, SDHC does go wrong, but if you are using totally unsuitable cards then you are going to crash and burn pretty quickly.

Sony have added the SDHC support. If you are using SDHC you need to be using a reputable adaptor with a reputable card and be running the latest firmware.

Remember people rarely post on here to say everything is going well, they only post when things go wrong. A great deal of the SDHC discussion on here as been generated as a result of user error or people trying to find cheap cards that work (the quick answer of course is cheap cards never work and you need to stick with the cards recommended by people like MxM).

I have to agree with the general thrust of this thread.

Using anything other than SxS cards for paid work is simply unprofessional in my view. Whilst you can never eliminate risk of failure somewhere in the chain using consumer grade media designed for stills is asking for trouble.

The cost of a couple of SxS cards set against a reshoot is trivial. Sure they're expensive but that's because they are orders of magnitude more reliable than SD or Memory Stick media. Photographers get away with them because they take many shots of the same thing and can afford to loose a few now and again.

Clients might never forgive you for playing data-loss roulette with their footage!

Whilst I agree that SXS cards are desirable, lets not put them on tooo much of a pedestal. For what they are, they're way over priced. Sony must be laughing all the way to the bank.

I Have one 8gb SXS card which came with the camera and I have used it once to try out overcranking. For all my other work I have been using SDHC cards and in 18 months have not had any problems - a gamble with pro work? No, no more than I used to have when I sent my 35mm, 120 roll, 5x4 sheet films off to a pro lab. I had many problems with various labs over the years. My EX3 so far has been very reliable, even in a hot and humid area of Africa. I have had several tapes chewed up in my Canon cameras, and ironically I only ever used Sony tapes.

I do check all my SD cards thouroughly before using them on an assignment. Yes, there may be a time when I am let down, but then that goes for any element on the camera too. I personally don't like having to thether the EX camera to a computer in order to download a SXS card, and the Sony card reader is also over the top price wise.

I'm sure this debate could go on for page after page SXS v SDHC, but the bottom line is that we have a choice and that choice may not be ideal, but may just reflect the users finances.

Amen. :-)

I think it's important to be clear that consumer media are not as reliable as SxS - despite many people reporting no problems.

I'm pretty risk adverse, so will only ever use the best media available, which at the moment is SxS.

You pays your money and you takes your choice :-)

I only use SxS media. I have not had any issues to date. Everyone's business is different. But in my market, a single-day shoot rate can be more than twice the cost of a single 140min 32GB SxS-1 Card. So, for the cost of half a shoot day, I get a quality card that I can reuse for years. Yes, it's possible that there can be a technical problem anywhere in the workflow, including with an SxS card. But I have been able to sustain a profitable business model without taking media shortcuts. (Even if the card pays for itself in 3-4 paid jobs for you, isn't it worth it for a greater piece of mind?) I agree that perhaps the SxS cards are overpriced. I suppose I can say the same thing about my EX3 (wishful thinking). But it didn't stop me from investing in the XDCAM-EX workflow. I'm all about trying to save costs, but having a good camera and good media isn't the place to take risks. I let my clients know about "take home" media options (if they want to walk away from a shoot with the media). Most of them appreciate the higher standards I have invested in.

The vast majority of SHDC problems posted on here are user error or people using the wrong cards. The user error part of the equation has been vastly reduced with the new firmware (with the corresponding decrease in problem posts) and the wrong card part has been reduced by people like MxM testing cards and identifying what types work well.

SxS has advantages. There is increased reliability but to claim it is in "orders of magnitude" is ludicrous. We've had plenty of people post on here with SxS problems.

But what about the advantage of premium SDHC cards such as the ATP? How about not having to format cards for reuse after you've offloaded them because you can afford to have more? In short even if you go for the premium SHDC cards the cost is still over a third less. You can afford to leave the cards sitting on the shelf for a few weeks because you have enough spare cards to be shooting on.

The issue here isn't SxS vs cheaper media. When he first used clipbrowser twice, he ended up with corrupted copies. On his next attempt using Explorer, his copied files were fine.

Marcus,

Would you mind pointing a few of them out, because I must have missed every one of them. I can't recall a single SxS problem being talked about that wasn't the result of operator error or a hardware problem with the camera itself. Apparently I have missed "plenty" of them. Could you point me to a few of the posts. Thanks.

Doug

The abiity to use Clip Browser (and FCP Transfer if you are a FCP owner) is critical to the XDCAM workflow. If you use cheaper cards that prevent the normal workflow, you have wasted valuable time, important metadata, and possibly even frames of video or entire clips. That's the issue.

Doug,

I'm sure you can figure out how to use the forum search feature but such a search quickly showed a few examples including one chap who had his SxS pack up 40 minutes into a card. The guy got his card replaced but it was scant compensation. Things can and do go wrong. I do know someone who spent some considerable time helping a colleague recover a corrupted clip from an SxS card.

We know your position on SDHC. That's fair enough, but shouting loudest isn't justification in itself. SDHC is a minefield and frankly some people shouldn't be using it because they don't understand what they are doing, but for those who can get the workflow sussed there's many benefits to be had including the ability to retain stock as if it were tape.

Sony recognise SDHC as a solution now and the firmware has been updated. We know which cards are to be trusted and which to avoid. It is a minefield if you don't know what you are doing.

And going back to the OP, I doubt SDHC had anything to do with it frankly. Just a bad transfer or some kind of glitch.

Does anyone know what the nature of the CRC checking is in Clip Browser? Windows and OSX will CRC check as a matter of course while copying files. I'm sure many of us have seen "a CRC error" on a hard disk that is failing. So what is Clip Browser doing that the OS doesn't do? From where I'm standing it's the job of the operating system to make sure that files copy correctly.

After all, how many times have you copied a file from one hard disk to another and found that it corrupted on the way across? In short, not in 25 years of being a computer user. Even my Sinclair Spectrum CRC checks data when loading from tape!

So what gives? Is this CRC check giving people false confidence? Is it checking what the OS would check anyway?

The files were reportedly recorded fine to the card and were not corrupt. This is confirmed by the fact that when he uses Windows Explorer to copy them, there are no playback problems.

This comes back to the original question of why the files are corrupting when using Clip Browser to transfer them - and even more troublesome is that the transfer is occurring with the CRC check in place.

So the logic is completely upside down in Ivan's report: the supposed "least safe way" to transfer - Windows Explorer - is actually proving to be the most reliable. While the supposed "most safe way" - Sony Clip Browser with CRC check - is failing.

I'm still unclear as to whether all the clips would play in the camera or not. His description was a bit confusing as to what would play and when. If they will all play okay in the camera, then the problem is downstream from there. However, that "Class 2" speed issue is still hanging out there too.

Don't the Sandisk cards say something like "Ultra 2" but then have the actual class rating in small writing? That might be causing confusion.

I'm sorry this thread has turned into another SxS vs. SDHC discussion - it was really not my intention, as you can see in my original post, my story ends with me asking if there is another program I could use instead Clip Browser.

Just for the record, I have been happily using 2 Class 4 and 1 Class 2 16GB Sandisk Ultra II cards with an e-films adaptor for a long time. I never had any problems except once I lost a little bit of a 48fps overcranked clip. I filled all three cards with 48fps footage and one clip failed, but I was able to recover almost all of it. I've been thinking of getting SXS-1 some day, but not now. Still too expensive for my taste, I don't shoot much paid stuff, I always state the 1% possibility of failure, and so on. I just bought 64GB in SDHC cards for 200 instead of 1200.

As some of you figured out already, my clips are fine. The problem was with Clip Browser. I'm sorry if this wasn't clear in my original post, it was never stated, but I thought it could be deducted. I promise for the sake of your curiosity I will run some tests, like, downloading the clips to another hard drive or downloading from the camera instead of using the card reader. This should help us narrow down the problem and see what we should watch out for in the future.

So windows explorer has a good built-in check? For copying and moving files.

I'm on a difficult position here, really. I even started using Clip Browser some time ago "to play it safe" - before I just copy-pasted with explorer - but imagine the consequences of checking a correctly copied clip, assuming everything is ok, and not editing the footage until some weeks later. I shoot quite often, so, there will be no chance to re-downloading anything...

*sigh*

Thats the bit I worry about too.
I make a folder on my external HDD and then sub folders for each of my sxs cards. I copy my SxS cards to the external HDD with C.Browser. Once all cards are copied I then copy and paste that folder from (with all the sub folders) the external HD to one on my PCs Hard drives.

I switch off the PC and the external HD. Start the PC again, bring the clips into the Edius timeline from my PC internal HD via CB.
I would look through the clips on the timeline making sure everything was there that I recorded.

Then I copy and paste that folder to another internal HD making 3 copies off that project saved to HDrives.

Allright guys, apparently the problem's my hard drive. Copying them to my other hard drive gives me no problems at all.

So I guess Clip Browser doesn't really verify the data in a "100% satisfaction" way. It would be nice to have a program that automatically checks the copied files not by just looking at numbers with a CRC, but actually somehow playing bits of it, frames for example, to check if there is any wrong or anomalous.

I HATE failing drives. Oh well, I guess I'll have to buy another one... again.

Moral of the story, Beevare of the big fat hard drive that lives beneath your case... he eats little boys, puppy dog tails, and, big, fat clips.

Having more than just one copy seems very reasonable - if not absolutely necessary. I would say checking the clips is very necessary as well, but the problem is still efficiency on this last subject. I just made a test and Premiere warns me about these bad clips when trying to conform them though, that's good for efficiency.

Thanks for sharing your workflow Anthony, it seems pretty flawless... I would like to ask if you don't mind, why the switch off and on step?

Thanks,

Iván.

Do you copy and paste using CB or Windows? I always use CB for everything that deals with footage. When I first got my EX1 I copied and pasted folders using Windows. Everything was fine for about a month. Then one day as I was editing, I noticed a dropout in my footage. It wasn't recoverable because I had already erased my SxS card. Since then, I've always used CB for transferring all files, every time. No problems since...that was about 3 years ago.

After using CB to copy my sxs cards to the external HD I just copy and paste (windows) the folder after that.

Perhaps I should use CB for everything.

Thanks.

Iván, could you check one clip in question to see if the file size (in bytes) of the file that you copied to your destination hard drive exactly matches the file size from your original media? I would suspect that Clip Browser doesn't report any copy errors if the file sizes are the same.

Another test would be to copy the problematic file to another physical drive - to confirm it's not some sort of performance issue with one specific drive. (Maybe the actual clip itself is fine - but the specific drive speed/bandwith is the issue- especially if it's failing)

Maybe it's possible that the original copy was in fact "ok", but some other corruption to the file occured after the transfer?

There wouldn't be anything else suddenly interfering with playback from the drive (virus checker, updated playback software, hard-drive setting change, etc)? You never know when some OS automatically updates some driver or something. (I keep mine disabled)

Well when I switch off my external HD Edius only reads from the copied files on my internal PC HD and if it reads them OK it proves to me that my original copy to my external HD was good. Well thats what I think anyway.

Hello Ed: the files are exactly the same size.

I finally found out it was a problem with my hard drive. I don't know if it's possible what you say about the file getting corrupt afterwards...

I don't think playback interference is the problem, because it's very determined, not random or sporadic or happening with every clip indistinctly.

Thanks!

Ivan.