Archival File Format - What's Best? at DVinfo.net

Go Back   DV Info Net > Sony XAVC / XDCAM / NXCAM / AVCHD / HDV / DV Camera Systems > Sony XDCAM EX Pro Handhelds

Sony XDCAM EX Pro Handhelds
Sony PMW-300, PXW-X200, PXW-X180 (back to EX3 & EX1) recording to SxS flash memory.


Reply
 
Thread Tools Search this Thread
Old February 23rd, 2008, 03:51 PM   #1
Regular Crew
 
Join Date: Feb 2008
Location: Champaign, IL
Posts: 32
Archival File Format - What's Best?

What is the best file format in which to archive EX1 footage?

My inclination is to store all data as .MOV files because I edit in FCP on the Mac. Rather than have archival files in the Sony MP4 format and a second file set in .MOV to store with the FCP files, it makes sense to me to simply trash the original .MP4 files and archive only the .MOV files.

My ideal archival system would have two sets of .MOV files. Set 1 would be all original footage simply converted from .MP4 to .MOV and archived. Set 2 would be only the .MOV files used with the project, and any related FCP files (titles, etc.).

In the world of still photography, the original RAW image files are kept in their original file format (which is proprietary for each camera manufacturer). This data is kept with all final output files (usually RGB TIFF) on optical disk and hard drive. The risk is that as RAW file converters change with the times, the original file formats may become unreadable (this is Adobe's motivation for the .DNG file format).

Any comments on this would be appreciated.
__________________
~ CB
Christopher Brown is offline   Reply With Quote
Old February 23rd, 2008, 04:32 PM   #2
Major Player
 
Join Date: Jan 2008
Location: Utrecht, NL | Europe 3rd Rock from the Sun
Posts: 612
Chris, I am also pondering archival methods...

You are correct in that 'everything computer' changes, so uncompressed video is likely to be the most cross-systems-and-time 'future proof' way. At a cost of diskspace and possibly disk speed issues in post.

I am considering an intermediate codec, specifically Cineform, to standardise on for post and archiving. If I was capturing some of the footage to Cineform directly (via HD SDI) that would make even more sense.

Although files would be around 3x as big as the EX1's mxf's, I'm not worried about diskspace as such, that is cheap enough.

I haven't figured out an economical way for archiving or duplicates (off site) for projects >>100GB. I would need qute a few BlueRay datadisks; maybe a (external/disconnected) harddisk would be a possibility. I'm not sure how reliable that last option would be though.

George/
George Kroonder is offline   Reply With Quote
Old February 23rd, 2008, 09:36 PM   #3
Major Player
 
Join Date: Jul 2007
Location: Richmond, VA
Posts: 229
My understanding is that if you trash the original MP4 data you make it more difficult to edit on other platforms and versions in the future.

My workflow at the moment is to drag the BPAV folder in the finder from SxS to an external drive. I then use XDCAM transfer to extract the .mov files to my internal RAID for editing, and back up the .mov files to DL-DVD.
Rob Collins is offline   Reply With Quote
Old February 23rd, 2008, 11:15 PM   #4
Major Player
 
Join Date: Sep 2007
Location: Palm Desert, California
Posts: 311
The BEST & ONLY format is the original! Keep clone copies of the .mpg4. Don't go archiving something that has been converted or processed. This is a basic general rule for all data security.
Michael H. Stevens is offline   Reply With Quote
Old February 24th, 2008, 12:53 AM   #5
Inner Circle
 
Join Date: Feb 2004
Location: Brooklyn, NY, USA
Posts: 3,782
If you hand somebody the XDCAM .MOV files created on a Mac with Sony XDCAM Transfer Tool, they can use MPEGStreamclip (even on Windows) to convert those files to another format.

Yes it's an extra step and yes it's not as simple has handing them a copy of the BPAV folder but it can work.
Craig Seeman is offline   Reply With Quote
Old February 24th, 2008, 03:46 AM   #6
Inner Circle
 
Join Date: Dec 2005
Location: Bracknell, Berkshire, UK
Posts: 4,957
You really should keep the original MP4/BPAV folders. The XML files contain information about when the footage was shot and some of the camera settings which could prove useful in the future. By saving the original files you will be able to work with the native material across many Mac and PC platforms without any processing. If you save as something else, then if you move to another platform you may have to re-encode or process the data which will lead to degradation. It doesn't matter how good the codec is, if you have to decode and the re-encode you will introduce errors (artifacts). These may be small and may be invisible at first but if you then have to convert to something else (DVD, Uncompressed, whatever) you will add errors on top of the errors from your first pass and very quickly you will see a degradation of picture quality. Concatenation is the enemy of video production and while adding an extra encoding stage at the beginning may not show any immediate visible effects you may end up regretting it later. Once you have added an extra encoding stage you can never get back to your original quality, so my advice is to always keep the native media in it's original state whenever possible.
__________________
Alister Chapman, Film-Maker/Stormchaser http://www.xdcam-user.com/alisters-blog/ My XDCAM site and blog. http://www.hurricane-rig.com
Alister Chapman is offline   Reply With Quote
Old February 24th, 2008, 07:46 AM   #7
Regular Crew
 
Join Date: Feb 2008
Location: Champaign, IL
Posts: 32
MP4 is the consensus, but what if...?

Thanks for the responses, I really appreciate it. The consensus is to keep the original MP4 data and convert it to whatever is needed for editing.

I agree, but there could come a time when the original data is not accessible. Keeping the MP4 files requires the use of Sony's Transfer utility, and this assumes that the program will be available from Sony in future versions that will run on new operating systems and new hardware.
__________________
~ CB
Christopher Brown is offline   Reply With Quote
Old February 24th, 2008, 07:58 AM   #8
Major Player
 
Join Date: Nov 2002
Location: Tokyo
Posts: 898
archives ...

http://www.bitjazz.com/en/products/sheervideo/

Graeme Nattress review on sheerVideo codec.

http://www.lafcpug.org/reviews/review_sheer_video.html
Dean Harrington is offline   Reply With Quote
Old February 24th, 2008, 09:08 AM   #9
Inner Circle
 
Join Date: Dec 2005
Location: Bracknell, Berkshire, UK
Posts: 4,957
But as I said no matter how good the codec by re-compressing you WILL introduce additional artifacts. Say you use the sheer codec, at the end of the edit you render back to MPEG2 for HD-DVD or Blu-Ray. Your material will have gone through at least 2 stages of encoding to and from different codecs. If you keep the MP4's you can go straight from the MP4 wrapped MPEG2 at 35Mb to you chosen bit rate of MPEG2 for output. 1 Pass, 1 encode step, only one codec so no additional errors introduced through cross codec conversion.
__________________
Alister Chapman, Film-Maker/Stormchaser http://www.xdcam-user.com/alisters-blog/ My XDCAM site and blog. http://www.hurricane-rig.com
Alister Chapman is offline   Reply With Quote
Old February 24th, 2008, 09:10 AM   #10
Wrangler
 
Join Date: Oct 2003
Location: DFW area, TX
Posts: 6,108
Images: 1
Quote:
Originally Posted by Chris Brown View Post
Thanks for the responses, I really appreciate it. The consensus is to keep the original MP4 data and convert it to whatever is needed for editing.

I agree, but there could come a time when the original data is not accessible. Keeping the MP4 files requires the use of Sony's Transfer utility, and this assumes that the program will be available from Sony in future versions that will run on new operating systems and new hardware.
I have to agree with Alister. Keep the original material. Sony has promised backwards compatibility with all future XDCAM products. You're going to see the XDCAM capability extended into higher bit rates in the future, but not abandoned altogether.

For robust archival, the XDCAM optical discs should last for 50 years.

-gb-
Greg Boston is offline   Reply With Quote
Old February 26th, 2008, 11:45 AM   #11
Major Player
 
Join Date: Oct 2001
Location: Washington D.C. Metro Area
Posts: 384
Quote:
Originally Posted by Alister Chapman View Post
But as I said no matter how good the codec by re-compressing you WILL introduce additional artifacts. Say you use the sheer codec, at the end of the edit you render back to MPEG2 for HD-DVD or Blu-Ray. Your material will have gone through at least 2 stages of encoding to and from different codecs. If you keep the MP4's you can go straight from the MP4 wrapped MPEG2 at 35Mb to you chosen bit rate of MPEG2 for output. 1 Pass, 1 encode step, only one codec so no additional errors introduced through cross codec conversion.
Alistair you didn't read up on the Sheer video codec did you?

It's mathematically lossless. As a result your arguments against using it are without merit.

That isn't to say I recommend it.

Essentially you are getting uncompressed video, but the codec adds another layer of technology to preserve in archival applications. In addition you "only" get a compression ratio slightly higher than 2:1, making the value proposition dubious.

Disk space will still be an issue with the sheer codec unfortunately, enough so that for intermediate archival purposes I still recommend uncompressed video.

For long term archival film remains the king... Do a filmout and have that properly stored.

For short term archival storing the acquisition master is a good solution. In this case copying the BPAV folder as has been suggested. The primary concern here is that eventually Sony will drop support for the XDCAM transfer software, either on your platform or in general. You must be prepared at that time to make other arrangements.
__________________
Alexander Ibrahim
http://www.alexanderibrahim.net
Alexander Ibrahim is offline   Reply
Reply

DV Info Net refers all where-to-buy and where-to-rent questions exclusively to these trusted full line dealers and rental houses...

Professional Video
(800) 833-4801
Portland, OR

B&H Photo Video
(866) 521-7381
New York, NY

Z.G.C.
(973) 335-4460
Mountain Lakes, NJ

Abel Cine Tech
(888) 700-4416
N.Y. NY & L.A. CA

Precision Camera
(800) 677-1023
Austin, TX

DV Info Net also encourages you to support local businesses and buy from an authorized dealer in your neighborhood.
  You are here: DV Info Net > Sony XAVC / XDCAM / NXCAM / AVCHD / HDV / DV Camera Systems > Sony XDCAM EX Pro Handhelds

Thread Tools Search this Thread
Search this Thread:

Advanced Search

 



Google
 

All times are GMT -6. The time now is 04:53 PM.


DV Info Net -- Real Names, Real People, Real Info!
1998-2017 The Digital Video Information Network