Dear Friends,

For years there has been a "Format War" between major camera manufacturers.

Camera manufacturers typically choose one side of the other: Intraframe or Long-GOP.

With all of the best intentions, and with their best engineers and designers, each camera manufacturer usually chose one camp or the other, and then stayed with it for many years. There are exceptions.

We try to remain fairly neutral in this "Format War".

This is very easy to do since we support both.

We do have our opinions, even strong opinions, but we feel that it is best to offer both choices, so that our users can choose whatever is best for their situation.

And, what is best for one shoot may not be best for the next one.

We have found that a lot of misinformation has been spread, and a lot of misconceptions still exist.


1. Some feel that when using Long-GOP, the only real, true frames are the "I-Frames" inside of the group of pictures.

For example, I was told that if a fast moving object did not appear in the initial I-Frame, inside of a group of pictures, that the object would not appear in the video.

The example was given that in a hockey game, a fast moving hockey puck could not be seen in Long-GOP.

I was able to quickly produce a video, demonstrating that Long-GOP handles fast motion, and the person was completely satisfied.

2. Another misconception is that one cannot edit a Long-GOP effectively since one can not cut unless on a Long-GOP boundary.

This is also false. The Long-GOP is fully decoded by the NLE into full frames, and one can cut on any frame desired.

3. Another strongly held belief is that Long-GOP is harder to edit, or editing system will be less responsive, a belief from the HDV days.

We are in a fairly unique situation in that our Long-GOP is always "Full Raster". For 1080, this means 1920 x 1080, even if a camera's sensor is 1440 x 1080.

This means that the NLE does not have to do any "heavy lifting" to work with our footage, in that the footage does not have to be converted to 1920 x 1080 from 1440 x 1080.


Since we support both Long-GOP and Intraframe equally well, I am just posting this to help clear up the misconceptions.

If one prefers to shoot in I-Frame Only (our version of Intraframe), then we fully support their choice.

The same applies to Long-GOP.

The selection of Long-GOP or I-Frame Only is very easy, it is just a menu selection in our Flash XDR or nanoFlash.

It does not mean that one has to switch camera vendors and purchase new cameras.

If one has an Intraframe camera and wants to shoot Long-GOP, our nanoFlash fully supports this choice.

If one wants to shoot Intraframe from a Long-GOP camera, this is also fully supported.

Hi Dan:
There is one aspect to this discourse which has thrown me for a loop mentally, and I'm hoping you can clarify this point for me Dan. My question lies in an earlier explanation you gave on this forum, whereby you stated that CD writes its I-Frame MPEG video in a Long-GOP of 15 frames. (??) OK. This blows my mind ! Why do you do it this way and can we really conclude the Flash XDR and Nano Flash are writing true I-Frame video structure, if you are writing the I-Frames as Long GOP structure ? Isn't this just Long GOP by another name ???

Respectfully,

Dear Mark,

We have always called our modes "Long-GOP" and "I-Frame Only".

Long-GOP starts with an I-Frame, and then has B and P Frames.

Our I-Frame Only is in a Long-GOP structure in which every frame is an I-Frame.
Every frame stands on its on for compression and decompression just as every Intraframe codec does.

This allows the editors that work with XDCam 50 Mbps 4:2:2 footage work with our "I-Frame Only".

Your Avid system should work well with our I-Frame Only footage, as should your Final Cut Pro system.

Hi Dan:
So you're stating that it's not CD's version of I-Frame per se, rather, this has always been the way I-Frame MPEG is produced ?

Mpeg 2 was originally developed as a long GoP codec. Long GoP codecs can have different combinations of I, B and P frames depending on their chosen use. In recent years it has become common practice to make use of I frames only and a simple way to do this is to have a GoP with nothing but I frames. The beauty of this is that you can use a single codec chip and matching codec software to work with both the traditional long GoP IBP type GoP's as well as the I frame GoP's. By doing this CD can make use of the XDCAM codecs installed in FCP, Avid etc to decode the I frame only codec without needing to write any additional software.

Dan, Mike.. correct me if I am wrong.

Recommended reading:

MPEG-2 - Wikipedia, the free encyclopedia (http://en.wikipedia.org/wiki/MPEG-2)

Dear Alister,

I fully agree with your post.

Hi Dan & Alister:
I thought there was a form of editable MPEG which did not use the Long GOP structure ?

2. Another misconception is that one cannot edit a Long-GOP effectively since one can not cut unless on a Long-GOP boundary.

This is also false. The Long-GOP is fully decoded by the NLE into full frames, and one can cut on any frame desired.


This misconception may actually be true regarding cutting and saving MPEG without recompression/rerendering.

Dear Gints,

It is my opinion, if one is using any of the modern Non-Linear Editors, the ones that are typically discussed on this forum, if a cut is made in the middle of a Group of Pictures, a new Group of Pictures will be created starting with the first frame after the cut.

To be fair, I do not know the details of how this is done. For example, I do not know if the NLE creates a smaller than normal Group of Pictures. and if there is no other reason to recompress and render. Or it could recreate the Group of Pictures until it gets to the end of this segment (source video segment).

All of this depends on how sophisticated the NLE is.

I assume that the most basic of editors, the very low cost editors, may do it differently.

Hi Dan:
In the latest iteration of Media Composer, Avid breaks up the entire Long GOP into a series of single frame MPEG's. I don't know how this is done, but I do know Matrox Digi-Suite was doing the same as early as 1997-98 is standard Definition. I know Avid has changed the way it handles MPEG once again, since version 3.5.x, because it is no longer necessary to make a video mixdown or transcode to output Long GOP based HDV projects. Now you can use a Quicktime Reference File to export to DVD or Web encoders, such as Sorensen Squeeze.

This misconception may actually be true regarding cutting and saving MPEG without recompression/rerendering.
NO.
Only the GOP that is cut is rebuild.
As Dan points, a smaller GOP is created.
GOPs don't necessarily have the same length.

The fully Intraframe structure of the NANO-I files can be tested with CinemaTool.
You can "Conform" NANO-Intra footage, while you can not conform NANO-LGOP footage.
A bit of reading would be very convenient.
Cheers,
Rafael

Hi Rafael:
You wrote: "NO.
Only the GOP that is cut is rebuild.
As Dan points, a smaller GOP is created.
GOPs don't necessarily have the same length.

The fully Intraframe structure of the NANO-I files can be tested with CinemaTool.
You can "Conform" NANO-Intra footage, while you can not conform NANO-LGOP footage.
A bit of reading would be very convenient.
Cheers,
Rafael"

.......Huh ?

Hi Dan:
In the latest iteration of Media Composer, Avid breaks up the entire Long GOP into a series of single frame MPEG's.

That's impossible as in the case of HDV when exporting to HDV tape or streaming over firewire to a deck the deck would not be able to decode the non-standard output. It also implies that everything is forced into a decompression-re-compression step with it's resultant loss of quality and I doubt Avid would force that upon it's users. Avid as far as I know still handles material natively, which is why AMA is able to work with footage straight of SxS cards or XDCAM discs without actually needing to import the footage. Matrox Digi-Suite used a pair of on-board hardware Mpeg codecs to decompress in real time for output over the SDi or Component connections. Any cuts or edits had to be rendered before you could export a Mpeg stream.
As Dan and others have said, only the GoP's immediately either side of any cuts or grades are processed. The way this is done seems to vary from NLE to NLE and may involve truncated GoPs. I have also heard of GoPs with frames of zero duration but never been able to confirm whether this is possible.

Hi Alister:
You wrote: "That's impossible as in the case of HDV when exporting to HDV tape or streaming over firewire to a deck the deck would not be able to decode the non-standard output. It also implies that everything is forced into a decompression-re-compression step with it's resultant loss of quality and I doubt Avid would force that upon it's users. Avid as far as I know still handles material natively, which is why AMA is able to work with footage straight of SxS cards or XDCAM discs without actually needing to import the footage. Matrox Digi-Suite used a pair of on-board hardware Mpeg codecs to decompress in real time for output over the SDi or Component connections. Any cuts or edits had to be rendered before you could export a Mpeg stream.
As Dan and others have said, only the GoP's immediately either side of any cuts or grades are processed. The way this is done seems to vary from NLE to NLE and may involve truncated GoPs. I have also heard of GoPs with frames of zero duration but never been able to confirm whether this is possible."

.......No, I'm not 100 % positive, but the new Avid is now totally format and resolution independent and I'm pretty sure it strips *ALL* MPEG of its Long GOP structure *without* transcoding it. I know Avid no longer transcodes HDV or XDCAM HD projects because they say in their literature it is no longer necessary to perform video mix-downs of an HDV or XDCAM sequence or clip in order to export it back out of the NLE. Somehow the new Media Composer (4.0.2) has the ability to see each piece of video (Any video) as individual frames, then re-assenble the Long GOP on the way back out-all in *real time !* Now this kind of data processing is a real accomplishment in programming. Alister, it doesn't matter where the cuts are made, because Avid will re-assemble the Long GOP structure appropriately. In fact, it may not even be as difficult as we think for the NLE to break down the Long GOP structure on the fly and then re-assemble it on the fly. I know it's CPU and GPU intensive, because the PC and MAC hardware requirements are hefty and strict. Avid only guarantees its software to run on approved machines in PC and MAC. The approved list is very short.

It is my opinion, if one is using any of the modern Non-Linear Editors, the ones that are typically discussed on this forum, if a cut is made in the middle of a Group of Pictures, a new Group of Pictures will be created starting with the first frame after the cut.


Dan, you are correct. I should have added that only *some* recompression would occur during the start and end cuts if either is not on a GOP boundary, but not the GOPs in-between.

Changing a codec from Long GoP to I frame only IS transcoding.

Avid doesn't need to transcode because it works natively with HDV and XDCAM, as I said it can edit material directly of the media, nothing is imported so it MUST be working with the native material in it's native codec.

The AMA import system works by using decoder scripts written by either Avid or the camera manufacturer. In the case of XDCAM the decoder script was written by Sony. The decoder plugin allows Avid to work with the material Natively.

Hi Mark,
You stated ...."I know Avid has changed the way it handles MPEG once again, since version 3.5.x, because it is no longer necessary to make a video mixdown or transcode to output Long GOP based HDV projects. Now you can use a Quicktime Reference File to export to DVD or Web encoders, such as Sorensen Squeeze."

This is not correct. You cannot export a quicktime reference of Long-GOP material in ANY version of Avid. As far as I know you cannot do this in any NLE. Apple's QT does not support the MPEG structure in a way that would allow this. This is not an Avid issue, but a Quicktime issue.

You may be confusing export of HDV material in some other fashion, but you CANNOT export a quicktime reference of Long-GOP material unless you first mixdown or transcode the material first.

The Avid forums have several current threads discussing this issue and I have done tests in Avid 4.0.2.

Hi Bob:

I'm quoting from the Avid Media Composer Readme ver 3.5x

"Long-GOP Media Creation
Avid editing applications now allow full media creation and playback support for all of the
specified Long-GOP formats and modes (XDCAM HD and XDCAM EX). This includes
support for video mixdown using Long-GOP HD formats."

....In the Avid forums they said this means the mixdown is performed automaticly and it is not necessary to transcode or mix it down automatically to export it out via QT Ref. I will have Avid Media Composer ver 4.0.2 this week and I will test it and tell you. EDIT: They specifically said HDV projects can now be directly exported in their native format via QT Ref without performing a video mixdown. Perhaps this is *Not* the case for the Sony Long GOP formats ?

Mark,
After detailed testing with Avid 4.0.2 and 4.0.3 and conversations with Avid there remains no way to export ANY Long-GOP file as a Quicktime reference file. Again, as I stated before this is due to Apple Quicktime not supporting this feature. Until Apple does something with Quicktime program you will never be able to export ANY Long-GOP file from Avid as a Quicktime reference file.

Steering this back to Convergent Designs, it does look promising that they will be able to provide support for many of their various recording codes that are not now supported in Avid through an AMA plugin.

Hi Bob:
You wrote: "Mark, After detailed testing with Avid 4.0.2 and 4.0.3 and conversations with Avid there remains no way to export ANY Long-GOP file as a Quicktime reference file. Again, as I stated before this is due to Apple Quicktime not supporting this feature. Until Apple does something with Quicktime program you will never be able to export ANY Long-GOP file from Avid as a Quicktime reference file."

.......Oh crap ! Well, this is what I was told from an editor who's opinion I trust. I suppose my good buddy got this info wrong. (??) Can AMC 4/0/3 output Long GOP as a full Quicktime Movie instead, Bob ?

Hi Mark,
This is not correct. You cannot export a quicktime reference of Long-GOP material in ANY version of Avid. As far as I know you cannot do this in any NLE. Apple's QT does not support the MPEG structure in a way that would allow this. This is not an Avid issue, but a Quicktime issue.

You may be confusing export of HDV material in some other fashion, but you CANNOT export a quicktime reference of Long-GOP material unless you first mixdown or transcode the material first.


I don't know in a PC, but in a MAC no problem to export Reference movies with XDCAM or NANO footage.
rafael

Hi Rafael:
Are you referring to Avid Media Composer running on a Mac, or are you referring to FCP ?

1. Some feel that when using Long-GOP, the only real, true frames are the "I-Frames" inside of the group of pictures.

For example, I was told that if a fast moving object did not appear in the initial I-Frame, inside of a group of pictures, that the object would not appear in the video.

The example was given that in a hockey game, a fast moving hockey puck could not be seen in Long-GOP.

I was able to quickly produce a video, demonstrating that Long-GOP handles fast motion, and the person was completely satisfied.

A funny way to put it -just to be clear, because of the I frame/predictive nature of long-GOP compression bit starved long-GOP codecs do suffer from poor motion rendering. For example, most of the cheaper handycams that use 19Mb/s or 24Mb/s fall apart in complex motion scenes or fast moving action. Even the standard 35Mb/s XDCamEx codec can fall apart. The nano and XDR use very high bitrate Long-GOP (in some cases higher than many intraframe codecs) so they render motion very, very well.

3. Another strongly held belief is that Long-GOP is harder to edit, or editing system will be less responsive, a belief from the HDV days.

We are in a fairly unique situation in that our Long-GOP is always "Full Raster". For 1080, this means 1920 x 1080, even if a camera's sensor is 1440 x 1080.

This means that the NLE does not have to do any "heavy lifting" to work with our footage, in that the footage does not have to be converted to 1920 x 1080 from 1440 x 1080.

True, full raster takes less computing power to decompress, but an NLE still has to decompress the GOP "on the fly" to individual frames, and recalculate and render either in the background or on the fly to new GOPs when the cut does not fall on an I frame. Clearly this takes more computing power than the intraframe codec. With todays modern PC's it only becomes an issue when you have multiple streams of video to decode. So Long-GOP definitely = less real time layers and effects. Just as compressed intraframe codecs mean less RT streams than uncompressed - this is less noticeable because you start to run into other botllenecks with uncompressed like hard drive throughput.

In practice would you notice it? Well I think you definitely would in multicam editing. Dan is right about one key point - it is easier than spatially compressed HDV1 which has the dual hit of MPEG long-GOP decompression and spatial decompression to handle. But not all HDV is spatially compressed - JVC's 720P HDV2 is full raster and actually uses only a 6 frame GOP - so technically it is easer to edit, but less visually pleasing:)

Don't get me wrong, I agree 100% with Dan's overall point - that these units are fantastically flexible, and get around the limitations of many cameras making them far more versatile tools.

Hi Rafael:
Are you referring to Avid Media Composer running on a Mac, or are you referring to FCP ?
Hi Mark,
I was talking about FC.
rafael

Dear John,

Thank you for posting.

I agree completely with all of your points. There are certainly cases where I-Frame Only should be used or considered.

We are scheduled to release 280 Mbps I-Frame Only to add to our 100/140/160 and 220 Mbps bit-rates that we already support.

But, overall, it is really hard to beat our 100 Mbps Long-GOP for many uses. Our users have the options to select what is best for their needs.



It would be nice if someone would run a test, using multiple streams of I-Frame Only at 220 or 280 Mbps (due soon) versus multiple streams of 100 Mbps Long-GOP.

While the computer always has less workload with I-Frame Only, the 2.2 to 2.8 times as much disk I/O (input/output operations) can also create problems.

One stream of 100 Mbps Long-GOP equals 12.5 Megabytes per second (ignoring overhead).
One stream of 220 Mbps I-Frame Only equals 27.5 Megabytes per second.
One stream of 280 Mbps I-Frame Only equals 35.0 Megabytes per second.

So, theoretically, one can have more streams of Long-GOP before reaching the throughput limit of one's disk subsystem. But, this does not become an actual problem, unless one reaches the throughput limit of one's disk subsystem.



John, I always enjoy your posts. It is obvious that you have a great deal of experience and expertise.

Dan,

That aspect is what initially sold me on XDCAM HD. It was the first to offer a tapeless workflow with the benefit of not requiring a dedicated capture card/RAID setup to work in HD. My lowly G5 1.8Ghz iMac could actually work with this stuff since the data rate was only slightly higher than DV25.

The trade-off has traditionally been...

More compression = Higher CPU load, less I/O overhead

Less compression = Lower CPU load, higher I/O overhead

The units offered by CD allow people to place the sweet spot of the trade-off where they need it by using different bit rates.

-gb-

I don't know in a PC, but in a MAC no problem to export Reference movies with XDCAM or NANO footage.
rafael

If the material is I-frame material that is true.

i.e. Pro-res material. FCP also has to convert the material before you can export a QT reference file.