Thanks all for the spirited discussion! I have purchased some inexpensive 1TB external drives and backed up my MFX files.

AVID Media Composer 2.7.7 and earlier wouldn't handle EX1 MXf. 3.0 does.

Perrone,

Thanks, that was incredibly clear and helpful.

Where does Apple Pro Res fit into this schema. Is it similar to Cineform?

Can you add a touch more that brings "Long GOP" into the picture as well?

- Lenny

I agree. With Clipbrowser, just drag the thumbnail to a folder. The BPAV structure goes with.

Thanks Perrone, incredibly clear and enlightening!

I work with broadcast news and archive the BPAV to RAID (one BPAV folder per story). My reason basically beeing I get these wonderful thumbnails, and all the metadata easy to read. Saves me a lot of time when I quickly need archive footage.

And in THIS case, I COMPLETELY agree that saving the BPAV folder makes total sense. Most people won't need the thumbnails in archiving their master files, but broadcast is a totally different animal.

Sure Leonard.

Apple's ProRes fits into phase 2 and phase 3. It's a robust codec that can stand up to editing (it was built for it) and is good enough to master in. There is also ProResHQ which is another step up, but is a bit "heavy" for most machines to use for editing. If you are cutting footage off RED or other "film" based footage it might make sense. ProRes is a bit lossy, but perfectly acceptable for a few generations of rendering.

The idea of long GOP is simple in theory.

Let's say we are shooting 30 progressive frames per second on our camera. It's on a tripod, and it's on a sidewalk aimed at a building. Every half second we record 15 frames of video. An intraframe camera like the HVX or like true film, will record 15 frames that are complete. It's like taking 15 rapid pictures with a digital still camera and that gets recorded to the media. In pictures that don't change much, this is an incredibly wasteful way to store data.

In the same scenario, Long GOP takes a full frame on frame 1. Then on frame 2 it takes another full frame and compares it to the first. It then only records the difference between the two and writes that to the media. It does the same for the next 13 frames. Well at least the Sony does because it uses a 15 frame GOP structure. Other cameras like the JVC use a 7 frame GOP structure which would handle rapid motion better.

Where Long GOP falls down is if there is a large amount of change between the frames in it's structure. At some point it can't keep up with how quickly things are changing. The software is usually pretty smart. It's making predictions of what will change from one frame to the next to speed up things. When that begins to fail, the picture quality begins to show artifacts. You can see exactly the same thing in your DVDs even if you start with intraframe sources because Mpeg2 which is used for DVDs (and the EX1) use the same technology.

Newer Long GOP codecs like the improved EX1 mpeg2 codec do a better job at this stuff than the older (DVD and HDV) mpeg2 based recorders. Better still are the new mpeg4/AVCHD recorders like the new Panasonic HMC150. The problem is that their efficiency at writing the data better comes at the cost of your computer working harder to DECODE the same stuff later. So you either need a faster computer for the new stuff, or you need to transcode to something easier to handle like Cineform or ProRes.

The internet community usually rails against Long GOP because it is harder on computers in editing, and the codec can be overwhelmed with fast action. What is nice is that the EX1/EX3 allows professional users and soon amateur users to bypass the long GOP codec. The SDI port on the side allows you to take the information off the sensors uncompressed. Just like in the $100K+ cameras. And you can record it any way you want. The downside is that you have to pay extra for that recorder. But in cameras without an SDI port, this option simply does not exist. Why this is exciting to pro users of the EX1/EX3 or amateurs with deeper pockets is that the EX1/EX3 give you the image sensors of a far more expensive camera, and with SDI based recording, you can now write that data stream. And you can do it for tens of thousands of dollars less than you could to do it 2 years ago.

Make sense?

Perrone,
Chapeau :-)

Thank's again Perrone, learning a lot here.

I also take my finished story cut in FCP and export it as XDcam MP4, import it to ClipBrowser and put it on my RAID. This way I can easily put it on the SxS-card in the camera, and play it out via the SDI.

If time is short I also use this way (SDI out) to feed a story via satellite instead of compressing and ftp.

As you point out, Perrone, Sony's long GOP codec is an interesting animal. At 35mbps(ala EX1), images are crystal clear, until you get full frame motion, like in a pan. Then the image breaks down, due to the compression artifacting at 35mbps.

While I still don't have my hands on a Convergent Design SDI to CF recorder(have a nanoflash on order)all the reports I've seen show a much reduced motion artifacting. The C-D CF recorders also use the Sony codec, storing data at a selectable 50 or 100mbps. This is a good compromise, since there's not a lot of portable equipment that can capture an HD-SDI signal.

Thanks again Perrone,

Your replies are now saved on my desktop. I sort of know most of this but it helps to see it all at once.
Now last questions -

1 - what's the diff between using Cineform or ProRes.

2 - I'm on a Macbook Pro - should I bypass ProRes HQ and just stick with regular ProRes?

3 - I know a lot of people who edit HD using Panasonic DVCPro100 codec. Often these are guys who are used to shooting P2 or with varicam etc. I've been trying to convince them to move over to an Ex-1 but they are freaked out by long GOP and the fact that its a different codec.

Can they convert Ex-1 XDCAM footage to DVCPro100?

I guess its kind of a pain for them because they need to convert all of it up front otherwise they are constantly rendering.

I've been told that the easiest way to cut EX-1 footage is to use and XDCam timeline but set for renders to be in ProRes. Can you do the same for DCPro100 and would that solve the problem? Just render once when dropping the stuff in and then it is always ProRes?

Sorry if this a bit afield of the rest of the thread here.

Lenny

Must be something wrong with your camera as my pictures don't break down on pans.

Perrone's statement that AVCHD is better than MPEG2 is not one I agree with. Certainly long GOP AVCHD can produce smaller files. AVCHD is a member of the Mpeg4 family and was really designed for very low bandwidth use such as web streaming and in that application Mpeg4 is very good, but I have yet to see any long GOP AVCHD or Mpeg 4 that has artifacts down at the level of XDCAM HD.

Perrone's description of the GOP sequence is simplified .Mpeg 2 not only forward predicts the motion in the picture but also backwards predicts, so your only ever 7 frames from your full reference frame, the Bi-directional predicted frames are call B frames, forward Predicted frames are P frames and the reference full frames are I frames. The frame sequence looks like this:
IBBPBBPBBPBBPBBIBBPBBPBBPBBPBBP

It could be the camera... or the operator. In my early days of film camera operating, we were told not to do pans of more then 30 degrees per second with the camera with a 90 degree angle to the picture pane. A 180 degree pan therefore took 6 seconds. To not to pan too fast we counted loudly... and it worked. Faster pans always resulted in visibly stuttering pictures on the screen.
That may be old craftmanship many an operator now smiles mildly upon, nevertheless video follows the same psychovisual laws as old perforated film (especially with progressive frame rates).

If I really need a fast pan, I either do it verrry slowly and speed that take up during post pro or I open the shutter as far as it gets with that specific camera, sometimes even adding a light horizontal motion blur in post pro before speeding the pan up to the desired value.

Leonard, I can't fake it. I know very little about editing on FCP. However, something you have pointed out here is one of the major reasons I never will. On to your questions:

1. Both Cineform and ProRes are terrific codecs. ProRes is apple specific and you cannot share data for editing with anyone on a PC using any program. Period. I am not certain if Cineform is PC only, but I've only seen it used on PCs.

2. Unless you are originating with very high quality footage (2k or higher), I'd just stick with ProRes.

3. DVCProHD is 1280x1080. Taking footage from a full raster camera like the EX1 to DVCProHD throws away a bunch of information. Your 1920x1080 becomes 1280x1080. The Panasonic guys edit in this codec because their cameras SHOOT in this codec. They don't record anything better in the first place, so no need to shift to a 1920x1080 codec in post.

The DVCPro HD codec is superior to the EX1 codec in a couple of ways. First it is intraframe and not long GOP. This has clear advantages during phase 1. Second, it captures marginally more color information per frame of footage. The big loss of the DVCPro HD codec is that it's horizontal resolution isn't nearly as good as XDCAM EX. And the XDCAM EX is terrible for phase 2 or 3. It's a phase 1 codec only (or primarily).

My beef with FCP is that you need to have all of your sequences match. I am SO used to Vegas where I can drop ANYTHING on the timeline, cut it, and render out. No need to convert anything at all. I have dropped EX1 footage, miniDV, and RED stuff (2k .mov) onto a timeline, cut it like it was native, and rendered out to Windows Media for streaming. It's all seamless. FCP really needs to get on the ball with this. For a pro level NLE, this is a really stifling limitation.

Fair enough. We can agree to disagree.

Watching mp4 encodes of 2k or 3k footage, with 40+ Mbps rates, I am inclined to disagree. I cannot speak to the footage quality of the HMC-150, but I have seen high bitrate AVCHD on BluRay, and it looks DARN good. Even with very high motion content.

Yes, I simplified. Thanks for filling in the blanks for a more full description.