Another Warning About The Locked Files Created by Nano/XDR
 

Hi guys. Hopefully none of you ever have as much trouble as I do with Convergent Design's decision to create "locked" files with the Nano and XDR recorders.

I know I've voiced my opinion in other threads about how broken this design decision is, but I thought it deserved it's own thread. Hopefully it will save you some headaches if you are aware of this.

The XDR and the Nano create "locked" files when you record them. CD decided that it was better to create locked files so that you would not delete them accidentally. I think they made this decision without thinking about the complete workflow.

First of all the Nano and XDR record to CF cards formatted with Fat32 filesystem. This is a filesystem that is native to Microsoft OS (Windows, etc). If you're working on a PC, you probably don't notice any special pains. You can connect the CF cards to your PC. Remove the "locked" attribute if you like, even delete the files if you want. You can't do any of this on the Mac.

Macs are able to read and write files to Fat32 but are not able to alter file attributes on Fat32 at all. So, there is no way to clear the locked attribute of the file on your CF card. You have to copy the files to your Mac, wait for the copy to complete and then remember to remove this locked attribute. If you copy the files from the card again to a different computer, don't forget to unlock them there too. In fact, you can't unlock they files on the card at all with a Mac. Either you get a PC and attach the card there, unlock the files for good or ... You can copy all the files to your Mac, unlock them, format the card and copy the unlocked files back to the card.

Then there's also the problem with pruning bad takes from your CF cards. Again, no way to do this with a Mac either. If you want to delete a file from your card, you'll need to connect the CF card to a PC. The Mac is unable to delete these locked files from Nano/XDR CF cards.

If you think all of this is a reason to really scratch your head on why they decided to do this, consider it even WORSE if you are using a Nexto-DI. I also purchased a Nexto DI from Convergent Designs to offload cards in the field. It was highly recommended by CD. Well you can probably see where this is going already, but the Mac is also unable to clear the locked archive on the Nexto. The Mac is unable to clean up the files or delete bad takes either.

If that all is not reason enough to re-think this really, really bad design decision, consider the FULL WEEK my editor just wasted because they were unaware that the FILES WERE LOCKED. My editor grabbed all the files from my Nexto after we wrapped for the day. I failed to warn the editor that the files were locked. The editor spent a few days working on the edit, which included setting AUX TC for all files (very time consuming). Everything looked good with the AUX TC and the edit until exiting FCP. The AUX TC disappeared on all the files after closing the project. After taking another whole day to re-enter the AUX TC on all the files, again the AUX TC vanished after re-opening the project.

As it turns out, FCP does not warn you when you set AUX TC on a file and FCP is unable to write to the files (because they are locked). Through the entire edit session, the aux tc was there but not being written to the file.

This was so frustrating and this edit was so far behind, I ended up doing the edit myself.

All these experiences with these locked files, leaves me scratching my head. If you're going to include such a frustrating and annoying "feature", at least give us a way to turn it off. Just for the record, I see absolutely no advantage to ever using this feature under any circumstance.

Aaron,

Sorry about the edit. Sounds very frustrating... and so unnecessary.

I agree with Aaron. For Mac users, this is a real pain. I think that the last time we discussed this Dan or someone from CD indicated that this wasn't hard to do and would be something they could include in the next rev. That begs the question: When's the next rev? It's been a little quiet around here lately which I'm taking as a good sign that the latest rev is working for most folks pretty well. I hate to unset the apple cart but this unlock fix would be most helpful.

Aaron, I found a way to batch unlock the files once they are on the Mac. I'll root around inside my edit machine and see if I can remember how I did that last... and post it.

Thanks!

Hi Aaron:
Man, I'm really, really shocked about this fact ! I had no warning and no idea about this. I know the Nano Flash and Flash XDR will never overwrite existing files already recorded, but I forgot about the implication of this on the MAC OS and its inability to modify the file attributes on FAT & FAT 32. Man, this really, really sucks ! I'm sorry to learn of your post agony. Looks like it's time to go Avid Media Composer 5. BTW, Avid Media Composer also runs perfectly on the MAC.

The real bug a boo I'm experiencing in post is if I shoot commercially, then everything has to be recorded in.MXF 50 Mbps to make the XDR files compatible with Avid TV Network post workflows. If I shoot in a high enough data rate to make using the XDR relevant in the first place to take advantage of its great quality increase (Read Recording in Long GOP 100 Mbps), then the resulting CD files are freaking useless to anyone in post ! They can't post the higher data rate files. If I stay in .MXF 50 Mbps Long GOP, then they ask why don't you simply use a Sony XDCAM HD camera and be done with it ? I tell them we can go all the way up to I-Frame intra 280 Mbps, but then they think that I can do Sony HDCAM SR - When I tell them it's still XDCAM HD 4:2:2 8 bit, then they don't understand. This is a conversation I really don't want to have to have with any folks at network. :-(

Now don't get me wrong, I still see great value in being able to take my prosumer HDV camcorder and record in TV Network ready Sony XDCAM HD 50 Mbps ready for post in Avid Media Composer, but the nano and the XDR have such great potential and superior performance when you push that built in hardware MPEG 2 Sony encoder to data rates of 220 Mbps and beyond - so naturally I want to go there. I've already requested a further elevation of the XDR data recording rate to at least approximate, if not equal, the lower HDCAM SR VTR mastering rate of 400 Mbps. I strenuously maintain a distinct visual difference between 220 and 280 Mbps on my HD monitor.

I just want get more network jobs with my gear man ! Avid Media Composer 5.0 will now handle Pro Res 422 and QT H.264 at any data rate, so this is a positive step in the right direction. The only problem with this is it will take ages before the Networks slowly upgrade to the next iteration of MC ! The other alternative is to *playout in realtime to HDCAM SR VTR tape at the station, but this plan goes south when they can't control the XDR from its RS-422 port ! It's not about the format, rather, it's about the workflow.

FileXaminer
 

Hey Aaron,

It's called FileXaminer from Gideon Softworks. FileXaminer It is a 14 day freebie... $10.00USD after that.

Select a bunch of locked files in the Finder. Open the program. Under Info, select "Batch Get Info on Finder Selection". Under the "General Information" dropdown, select "Advanced". Highlight all the files in the left column. Select "Unlock Files". Hit "Apply Advanced"... and voila!

HTH,

Automator
 

Aaron, I feel your pain and frustration... Very sorry about the time and lost of edits a few times.
I have posted a few times about an "Automator". I have done my workflows and a right click on my Mighty Mouse I have a lot of options to choose from. I have never experience any trouble of not able of reading, edit, delete my NF files. I use MBP 17" 2.4 Ghz Core 2 Duo.
I do not use Nexto-DI. I have enough CF cards to work 48h at 180 without repeating a card, so I really do not have a solution for you there, but, for CF cards or any external HDD "Automator" works magically well. Now I have to fine sometime to split the the audio from mono in QT to stereo....
If help need it just PM.
Cheers

Gentlemen,

Just read your thread so I decided to do a test on my editing suite Mac. It is running Mac OS 10.5.8 (aka Leopard). I selected all the nanoflash files in a directory on an external RAID, invoked a Command-I for get info, it opened up a "Multiple Item Info" window, I unchecked "locked" and all files stayed unlocked.

Possible other version of OS 10 won't let you do this or maybe if the files are on your root drive. Someone else will have to confirm this.

The files I just unlocked are RAW nanoFlash files.

-Andrew

That is correct Andrew!. I use the same Operating System. I use "Automator" because of other functions combine with unlock (like -rename, copy to backup HDD, copy to FCP Edit HDD, take a snap shot picture and crop and add to DVD for producer/director, or upload to FTP.... and more) great app

There are lots of ways to unlock the files on a Mac.

The easiest is just to put all the files in one folder, then get info on that folder (command-I), scroll to the bottom of the info window and change the privileges for everyone to Read and Write, you may need to authenticate (login and password for an admin user).

Then check apply to inclosed items in the menu below, the "cog wheel".

It may seem a pain but you get fast at it quickly. Or you can automate as in above posts.

You guys are talking about unlocking files ON THE MAC. I know how to do that. I agree, it's very easy to do if you know they are locked. My editor didn't know this. Once you copy the locked files to the Mac, they are easy to unlock.

My gripe is that there is no way to unlock the files ON THE CF CARD or on the Nexto. This is a problem. You can't prune the CF cards from a Mac. You can't keep your Nexto tidy from a Mac either.

agreed, it seems like a pain in the butt. I can't trim shots when I download from cards in the field.

Hello Aron,
I am sorry to hear about your issues you are having with locked files it must be very frustrating for you as others have pointed out it is really so easy to unlock files on the Mac. I use FCP and I can unlock the whole directory of CF files in five key strokes. I vote to keep the files locked and secure in case of accidental damage.

Andrew... very smart. I'm running the same system. With a little help from "Mac Help", I discovered that selecting multiple items, then holding down the Control Key and clicking on those items, produces a "Get Summary Info" selection where "Get Info" normally appears. Select that and one window appears that let's you change the common attributes of all the selected files. Sweet! And I wasted $10.00...

Back to Aaron's point, having a nanoFlash menu item that let's the user decide whether to Lock or Unlock files seems to be the answer here. Dan?

Dear Friends,

We are making changes in this area.

I have delayed posting as I wanted to get the latest, accurate information from Tommy.

As soon as I do, I will post what we are doing.

Good News
 

Hi Dan:
This is good to learn of. I sincerely hope this isn't the only development awaiting us XDR and Nano users.

Dear Friends,

Locking of the files does not appear to be an operational problem on PC's, but is a problem, under certain circumstances on a Mac.

We have made changes in our code, which will be in our next release.

This will remove all locking of the files.

This is not a perfect solution, as some prefer for us to lock the files.

We are striving to keep our menu structure simple. There is a price to pay if we have options for everything: the menu's get complicated and the device becomes intimidating, which we want to avoid.

If someone accidently deletes a file, while it is on a PC or a Mac disk drive, it will go into the deleted files folder.

you're able to unlock files on a fat32 formatted cf card on a mac? Without downloading the files from the card? You unlock them while they are still on the cf card. Can you give me a step by step of how you do this?

Hello Aaron,
I only unlock the files once they are copied over to the Mac. I open the folder and select all then unlock the 1st file and press the change all button then all the files are unlocked. I have not been able to change the sharing & permissions of the card to delete selected files from the CF card. This has not been an issue for me as I always carry extra cards while shooting and never delete or format cards on location. I wait until I return to base and have made a copy of the cards to my raid 1 drives before doing anything. That's just my work flow to overcome accidental erasure in the field.

Dear Friends,

I just want to emphasize that we do not support deleting files on the CompactFlash card, then re-inserting the card into the Flash XDR or nanoFlash.

Instead, we request that the card be Formatted, in the unit, prior to recording.

The goal is to prevent file fragmentation, which severely slows down the write process.

Deleting files then re-inserting and using the card, without a Format in the Flash XDR or nanoFlash, may cause file corruption.

No Re-Insertion Intended
 

...Hi Dan: Yes, true enough, however, I don't think it's the intention of anyone here to want to re-insert and then continue recording after turning off a locked file option. The whole premise here is that the cards would have all been blank, formatted in the Flash XDR or Nano ahead of time, recorded in the units *unlocked,* then removed and taken straight to post. (Aaron, correct me if I'm wrong here). I think we all have an intrinsic understanding that if such an option was available to us that we would not be heading back to the Nano or Flash XDR to continue shooting again without a total format *within the recorders themselves.

Dear Mark,

I did not want anyone to mis-read the following quote and think that deletion of files from a card is an acceptable procedure. We discuss this in our manual, however not everyone reads the manual.

A Clarification
 

Hi Dan: Understood. No one should be trying to delete files from CF cards, then stick the cards back into a Nano or XDR and start shooting again without an entire format.

*However, I don't think this is the issue Aaron is discussing here at all - The issue is about recording locked files on the CF Cards themselves within the XDR or Nano and his agonies in post as the result of being unable to unlock the resulting recorded nano or XDR files to go into FCP.

EDIT: Not being able to unlock the files from the CF media itself

Dear Mark,

In Post #15, above, the last post on Page 1 of this thread, I stated that we are removing "all locking of files" in the next firmware release.

Good News
 

....Hi Dan: My apologies. I must have accidentally skipped past this particular post. OK. I'll go back and re-read. Sorry.

Well the same goes for the Nexto DI then too. Scratch what I said about deleting files from CF cards since it's not supported. I can't delete files from the Nexto because of this silly locking issue.

Dear Aaron,

Please help me understand your current feelings.

We have offered to remove all file locking in future firmware releases.

I feel that this would solve the problem for you, for both the nanoFlash's files, and for the files when they are on the Nexto.

Is there something I am missing?

Hi Dan. Removing the locking feature or at least allowing users to turn it off WILL solve the problem.

What makes no sense is that the replies are leaning toward, "there is no problem" which isn't true. Those are where my replies are directed.

Once I'm able to record on the XDR without the files being locked, then this issue is closed for me and we can even lock this thread if you like. I have no idea when that will actually be though, if it is buried in the responses above, then sorry I missed it.

Dear Aaron,

The removal of "file locking" has already been implemented and tested, it will be in the next release for both the nanoFlash and the Flash XDR.

We hope to release the next firmware in approximately two weeks or so.

Encouraging Update Dan :-)
 

...Hi Dan: Well, this reads as encouraging news. I look forward to the next release and I also hope other bug fixes, such as .MPG playback, Canon 24 F Jam Sync, and perhaps a possible higher data rate record option for I-Frame recording (???) may be there as well :-) You wrote in my other thread last week that you folks at CD were working on some stuff you thought I would like. I'm hoping for all the best Dan. This is all an end user ever hopes for :-) Thanks for your patience. I'm actually not trying to piss you off, but reporting is often a negative sort of perspective. (i.e. Always harping on what's *Not working). Sorry, can't hep that-Please expect more of the same ;-) Don't shoot beta messenger Mark ! :-) ;-)

Dear Mark,

We have made signifcant "under the hood" enhacements.

And we have made many visible enhancements:

1. 24p Sync should work for you now. This has been tested in Japan and they found it to be perfect.

2. A visible Red "Recording" bar at the bottom of any attached monitor while we are recording.
This is enabled via a menu option.

3. File Locking has been removed.

I believe that MPG playback has been fixed and there are many other enhancements, I just can not remember them all at this time.

Our top bit-rate for this release will be 280 Mbps I-Frame Only.

Cool ! Cool ! Cool !
 

Hi Dan:
Looks very encouraging from here. I look forward to the new release.

[QUOTE=Aaron Newsome;1528076]
Then there's also the problem with pruning bad takes from your CF cards. Again, no way to do this with a Mac either. If you want to delete a file from your card, you'll need to connect the CF card to a PC. The Mac is unable to delete these locked files from Nano/XDR CF cards.

Sorry Guy's the way I read it sounds like editing the card on location.

Dan I do remember that there was mention that CD were looking at implementing the deletion of the last file on the cards is that feature still listed for implementation.

Dear Lance,

Yes, we will have "Delete Last Clip".

If you record a clip, then want to delete it, you can.

If you power down the nanoFlash, or remove the CompactFlash card, then you can no longer delete the last clip.

We can delete the last clip since we do it in such a way to prevent any possibility of file system fragmentation.

Nope, that does not work. I mean you can do it, but then you can no longer use the card in the nF without reformating it. Which really makes the entire nF much less useful as that means you need to carry a number of spare cards on you everywhere you go.

They claim that if they allowed it, it would slow the writing process down. I am not buying into that, except if you did it numerous times. Other cameras allow you to delete files in the middle just fine. It makes me think the nF does not have enough memory available.

Dear Adam,

When one deletes a file, it leaves an empty space.

In order to be able to write high bit-rate video to the card, we need to write very large blocks of data at one time. We do not have time to determine where the empty areas are on the CompactFlash card.

The way we do it, to make this all work, is that from the last file on the card, to the end of the card, is always free. Thus, we can delete the last file (clip), since this just makes this one big empty area (free space) bigger. It does not create two empty spaces, one in the middle and one at the end.

It is always best to degragment your computer files for maximum performance.

What we do, is keep our file system (the CompactFlash card), defragmented at all times.

For a stills camera, it is not a big deal to write a single, or a few images to the available space on the card, whereever that might be.

It is something else entirely to write high bit-rate data, for a long period of time, where one can not afford any wasted time.

Adam, we have a tremendous amount of memory in the nanoFlash. We use quite a lot of it to buffer the writes to the CompactFlash cards. This large buffer allows us to stop writing to one file (when we approach our file size limit), then open another file, then go back a close the previous file, writing out a very large index of every frame in that file, all the while we are writing the new data that is coming in to the new file.

Adam, the only reason I brought this up is because the last time I had to "hand off cards to the producer", I was left a little bit frustrated that I could not "clean the garbage shots off the card".

The producer got my cards,.. well actually, their cards since they now own them. They really don't care that they can't be played back in the nano but I cared that they have files where I accidentally recorded 20 seconds of the floor and my feet.

If I would have had a PC on the set where I could delete this stuff off the cards, I definitely would have.

Yes, I see what you mean. Makes perfect sense. I am glad for you (and other Mac users) that they are unlocking the files in the next firmware revision.

Make File Unlocking An Option
 

Hi Aaron:
I am a little concerned exactly *how CD will implement this. I hope they implement this feature as a feature (Read - Option in the menu) as opposed to it's either this way/or the other way, but not *and.

I've had some thoughts on this subject but I wanted to mull this over as there are a lot of concerns to be addressed that go well beyond the personal needs or desires of the camera operator.

In short I believe Convergent Design did the right thing given the history and protocol around the treatment of source "footage" coming off a camera. At present, we have had about a generation of use with solid state media in the industry with the most prevalent being P2 media or at least the one which paved the way. Prior to that it was tape and film as the predominant medium. The viewpoint of source media is that it is RAW and you are not supposed to mess with it. Insuring productions has been based around this premise and protocols in workflow have been created to ensure the integrity of source material. What some are asking for is a deviation away from this standard which is fine but you are referring to it as a flaw. It isn't if you buy into the history on the matter. I can see why people want an option to be able to mess with the source material but you are asking to do it on the transport medium.

Convergent Design has it's own requirements for people not messing with the cards but in terms of the actual argument about messing with source files has a great deal of precedent. I do buy into the idea that a shooter may want to clean up his/her footage before releasing it to the client, like shooting your foot (we've all done it) but with tape and film you can't do that, all of your sins were enshrined on the source medium. Convergent Design has made it very clear to us on the forum (and in the user manual) that the cards, the transport medium should be handled and manipulated only by the unit. You transfer the files to your nexto or your computer and eject the card without doing anything to it save a simple copy. Following this procedure allows them to push the abilities of the media to it's limits with us being the beneficiaries of this "bleeding edge" technology.

I would hope that Convergent Design implements this feature, as Mark suggests, so it is an option and it will satisfy the insurance companies that have been reluctant to get in the business of insuring "footage" that comes from a solid state medium. This matter goes way beyond personal needs.

Dear Friends,

We have listened to the points and requests on both sides.

We have decided to implement "File Locking" as an menu option in a future release.

Today, we released our latest code to our Quality Control tester. The next release is thus in test at this moment. This next release, most likely, will have "File Locking" turned off, without a menu option.

As soon as humanly possible, but without delaying the release of our next firmware, we will be adding "File Locking" as a menu option.

We have a great many features and internal improvements in this next release and we do not want to delay it. We hope everyone understands.

The Real Culprit is Final Cut Pro !!!
 

.....No it isn't, but the principal reason I understood why Aaron was requesting a lock turned off option was to suit the editorial eccentricities of Final Cut Pro's Auxiliary Time Code logging function. If CF Card media is locked, then FCP doesn't write Auxiliary Time Code to the proper file, so when you re-open your FCP Project this information vanishes ! You may absolutely require the Source Auxiliary Time Code from your XDR or Nano recorded video files to re-sync your double system audio. I know Aaron is shooting with his Viper camera, which doesn't possess any internal recording VTR, so he's either shooting double system sound or capturing dialogue audio on his XDR directly.

.....Yes, there is a direct *Legal implication here. The way the laws are written in Canada and particularly in the US, it's litigation city on CD from insurance companies (Read Film Guarantors) if an XDR or Nano has this option turned off ! If CD implements it as a fully visible menu option, then they are covered in a court of law because the end user has to make a *CHOICE* to implement it ;-) This is how the law would view this issue.