AVCHD - OSX workflow for simple re-encoding at DVinfo.net

Justin Beech · August 5th, 2010, 04:09 PM

Am I missing something.

My "HQ" AVCHD clips (1080i) are about 500mb for 10 minutes, with 5.1 audio.

Importing into Final Cut Express with Log and Transfer runs ok but then I must add the "interlace" filter, and "Render", before I can "Export". The "Render" phase for 10 minutes of footage takes several hours (4 core iMac 3ghz). Once "rendered" then "export" at 720p takes, I don't know, 10 or 20 minutes -- anyway the problem is "Render" taking several hours.

Using ffmpeg with -hq settings on the command line, the entire pipeline (read MTS, de-interlace, down-mix audio to 2 channel stereo, re-encode h-264 at 720p) runs at about 10fps, so about 30 minutes.

So from where I stand it seems that Final Cut Express is *not* a solution, unless there is something badly wrong with my use of it, I can't believe anyone uses it in this state for AVCHD and wanting h-264 output. Maybe I am missing something in the setup or configuration that magically makes it usable?

Are there any other solutions under OSX proven to work ok? I had no luck using any others. I tried VoltaicHD, I tried JES de-interlacer (it created a 150gb temporary file then churned away after the de-interlacing trying to export the 720p clip). Toast Titanium would not de-interlace. (since it uses the same output dialog boxes as Final Cut Express for settings, and the de-interlace checkbox on the quicktime output filter is useless, that may not be its fault).

Michael Wisniewski · August 5th, 2010, 07:58 PM

Try applying the de-interlace video filter to each clip and see if that speeds up the initial render.

Justin Beech · August 5th, 2010, 09:16 PM

I only have one clip in this example? it is a 10 minute long video, so start FCE with new project after log/transfer (or import 500mb MTS file - doesn't matter really), I drag it into the timeline, get a dialog saying something about FCE will change the format of the sequence to match, add the filter, then "render" and sit back and initially it predicts 5 hours, but takes maybe 3-4 depends what else is going on on the machine. It takes 150% cpu during this time. Then the bars above the time line are all red, and I am free to edit..

obviously it is impractical for me to import or log & transfer 100 MTS files and deal with each this way.

Tom Cadwalader · August 6th, 2010, 09:59 AM

Justin,
I don't think you should need to render after log and transfer. I don't.
Have you updated FCE?
Do your project settings, easy setup and your camera settings match. This seems like to most likely issue.
It sounds like the render is converting your material to match your timeline settings.
I log and transfer 1920 x 1080 60 i avchd to a project with frame rates and frame sizes matched.
Another issue could be the audio settings. My camera can record 5.1 but I use the stereo settings. FCE does not work with 5.1 audio.
To get form interlaced to progressive (for web or computer viewing) I do that on export. (QT conversion)
Hope I have helped.

Justin Beech · August 7th, 2010, 01:54 AM

I'll try that checking settings, thanks.

edit: what am I supposed to be looking at.. The only screen that seems to have applicable items is "Easy Setup.." there I can pick 'Format (all formats) Rate (all rates) and Use: AVCHD - AIC 1920x1080i60 ?

thanks.

Tom Cadwalader · August 7th, 2010, 08:50 PM

Yes this is where you set the sequence settings. They need to match what your camera was set for....to avoid a render.
I also recall an issue with the audio, if stereo was not selected somewhere ( I can't recall where) there were issues. You said you recorded 5.1, FCE doesn't work with that and it needs to convert it to stereo somewhere in the process.

Justin Beech · August 9th, 2010, 12:55 AM

results:

File from MTS via ClipWrap: 2.47gb, uncompressed audio (audio is the majority of the file).
"H.264, 1920 x 1080, Millions AAC, 2 channels, 44100 Hz" (less than 9 minutes of footage)

After setup for h-264, and New Project..
dragging ClipWrap 9 minute .mov file to FCE: instant
Adding 'De-interlace' filter: instant
Doing Render:
Phase 1: Writing Video .. (Estimated Time: About 3 hours) (started at 10:18am)
During this time, CPU usage is about 140% and disk activity comes in short bursts of about 500k/sec
11:35am - finished (1 hour 25 minutes)
then .. Export as 720p and 5000kbit .. start at 11:50am, estimated time "about 45 minutes"
During this time, CPU usage is about 150%

Total time: 2h 30m including running ClipWrap first.

vs ffmpeg,
ffmpeg -i 00000.MTS -deinterlace -vcodec libx264 -vpre hq -s hd720 -g 30 -b 5000k -ac 2 -ar 48000 -ab 320k -threads 0 test.mov

take native MTS file, converts to same .mov in 9 frames per second - 30 minutes. 180% cpu very small disk activity and low memory footprint.

so does this sound about right? 2h 30m per 9 minutes of footage for Final Cut Express?

Tom Cadwalader · August 9th, 2010, 08:01 AM

You don't mention any editing. So if you are not adding complicated titles, transitions etc, I don't believe you should need to render at all, except for the filter you applied. I suspect that the red bar for needing render comes on after you apply the de-interlace filter. If you are de-interlacing the entire 9 minutes that must be what it's rendering. I guess I would ask why you are applying that filter in the first place. This filter and Flicker are normally used on freeze frames or portions of clips with thin horizontal lines are on the screen the have a shimmer, or a still shot that has key framed movement. It sounds like you are de- interlacing the entire movie.
When does the red bar appear on the top of the timeline?
I am a little confused about your work flow. I think you are starting with 1920 x 1080 60 i AVCHD from your camera and you want a quicktime movie at h.264 720p for uploading to the internet perhaps.
What I have done is log and transer my 1920 x 1080 60i(not 5.1 audio) into FCE> edit> render> mixdown the audio> export to Quicktime conversion set for h.264 with the settings and any filters I desire. The conversion does take some time ( hour or two) if it is self contained. Very quick if not self contained. I get it going and then go do something else. Works great for me.
Dance & Sing Under Wings By Thomas J Cadwalader On ExposureRoom

You mention H.264 for your setup. Your easy set up should be 1920 x 1080 Apple Intermediate Codec. (Not the HDV settings.) If you convert MTS. files they should also be AIC, if you are converting to H.264, these clips will need to be rendered when dropped into the timeline as FCE will not edit H.264 without rendering, if at all. I haven't tried this.
I have used MPEGStream clip to convert from H.264 to AIC, edited, then exported back to H.264, that works pretty well, but I think you are starting with AVCHD.

Guy McLoughlin · August 9th, 2010, 09:00 AM

Make sure you are rendering to a different physical drive. Rending to the same drive that the AVCHD assets are on can take as much as 10x longer.

Tom Cadwalader · August 9th, 2010, 12:12 PM

Good Point, this can make a huge difference.

Justin Beech · August 9th, 2010, 03:15 PM

I'll try all hints in the last few posts, today, and see where that gets me. thanks.
I do have 'scratch' drive set to an external drive, but yeah, I'll experiment some more.

the de-interlace filter is necessary because the movie has fast motion .. nothing else gets rid of blurred fast-moving borders of high contrast that result from re-encoding these interlaced areas. We're not talking just a little comb effect on the edge of some things.

Justin Beech · August 10th, 2010, 05:10 AM

well, the results were not good.

New Project,
Drag .mov file into browser (9 minutes 1080i source material)
Drag .mov file from Browser to otherwise empty timeline
Add Filter .. de-interlace
(red lines appear, but I do not "Render")
Select Project again in Browser
Export .. Quicktime Conversion .. (720p) .. the estimated time is 4 hours.

Cancel that and remove the de-interlace filter
Export .. Quicktime Conversion .. the estimated time is still 4 hours.

Note, I'm writing to my home directory, and the FCE scratch disk is setup as an external drive.

Tom Cadwalader · August 10th, 2010, 06:32 AM

Well it looks like the de-interlace for the entire clip is what is taking so long. FCE has to create 9 mins of video from your source material. The more movement the more work it has to do.
I would offer 2 suggestions.
1) Do the de-interlace during the conversion to Quicktime and see what that looks like.
2) If you don't need a stand alone copy of the quicktime file, uncheck self contained. This will speed up the conversion from hours to minutes.
I would test things with a shot segment of the movie. You could copy a <1 min segment with a lot of whatever you are trying to overcome, and drop that into a new sequence. This will let you try somethings, but not spend hours to see the results.

Justin Beech · August 11th, 2010, 01:48 AM

I think I'm giving up on FCE for conversion of h-264 HD input ..

Checking "de-interlace" in "quicktime conversion.." does not, and I didn't expect it to, because it also doesn't de-interlace when checked during exporting from Toast 9.

On top of that, just doing the Export from FCE without any filters, just the quicktime conversion settings, STILL took several hours to run..

Tom Cadwalader · August 12th, 2010, 08:51 AM

Justin,
Maybe I should have asked this first, what camera are you shooting with?

August 5th, 2010, 04:09 PM	#1
Justin Beech New Boot Join Date: Aug 2010 Location: Sydney, Australia Posts: 9	AVCHD - OSX workflow for simple re-encoding Am I missing something. My "HQ" AVCHD clips (1080i) are about 500mb for 10 minutes, with 5.1 audio. Importing into Final Cut Express with Log and Transfer runs ok but then I must add the "interlace" filter, and "Render", before I can "Export". The "Render" phase for 10 minutes of footage takes several hours (4 core iMac 3ghz). Once "rendered" then "export" at 720p takes, I don't know, 10 or 20 minutes -- anyway the problem is "Render" taking several hours. Using ffmpeg with -hq settings on the command line, the entire pipeline (read MTS, de-interlace, down-mix audio to 2 channel stereo, re-encode h-264 at 720p) runs at about 10fps, so about 30 minutes. So from where I stand it seems that Final Cut Express is not a solution, unless there is something badly wrong with my use of it, I can't believe anyone uses it in this state for AVCHD and wanting h-264 output. Maybe I am missing something in the setup or configuration that magically makes it usable? Are there any other solutions under OSX proven to work ok? I had no luck using any others. I tried VoltaicHD, I tried JES de-interlacer (it created a 150gb temporary file then churned away after the de-interlacing trying to export the 720p clip). Toast Titanium would not de-interlace. (since it uses the same output dialog boxes as Final Cut Express for settings, and the de-interlace checkbox on the quicktime output filter is useless, that may not be its fault).

August 6th, 2010, 09:59 AM	#4
Tom Cadwalader Regular Crew Join Date: Jun 2008 Location: Cheboygan, Michigan Posts: 79	Setting not right? Justin, I don't think you should need to render after log and transfer. I don't. Have you updated FCE? Do your project settings, easy setup and your camera settings match. This seems like to most likely issue. It sounds like the render is converting your material to match your timeline settings. I log and transfer 1920 x 1080 60 i avchd to a project with frame rates and frame sizes matched. Another issue could be the audio settings. My camera can record 5.1 but I use the stereo settings. FCE does not work with 5.1 audio. To get form interlaced to progressive (for web or computer viewing) I do that on export. (QT conversion) Hope I have helped.

August 9th, 2010, 08:01 AM	#8
Tom Cadwalader Regular Crew Join Date: Jun 2008 Location: Cheboygan, Michigan Posts: 79	Maybe You don't mention any editing. So if you are not adding complicated titles, transitions etc, I don't believe you should need to render at all, except for the filter you applied. I suspect that the red bar for needing render comes on after you apply the de-interlace filter. If you are de-interlacing the entire 9 minutes that must be what it's rendering. I guess I would ask why you are applying that filter in the first place. This filter and Flicker are normally used on freeze frames or portions of clips with thin horizontal lines are on the screen the have a shimmer, or a still shot that has key framed movement. It sounds like you are de- interlacing the entire movie. When does the red bar appear on the top of the timeline? I am a little confused about your work flow. I think you are starting with 1920 x 1080 60 i AVCHD from your camera and you want a quicktime movie at h.264 720p for uploading to the internet perhaps. What I have done is log and transer my 1920 x 1080 60i(not 5.1 audio) into FCE> edit> render> mixdown the audio> export to Quicktime conversion set for h.264 with the settings and any filters I desire. The conversion does take some time ( hour or two) if it is self contained. Very quick if not self contained. I get it going and then go do something else. Works great for me. Dance & Sing Under Wings By Thomas J Cadwalader On ExposureRoom You mention H.264 for your setup. Your easy set up should be 1920 x 1080 Apple Intermediate Codec. (Not the HDV settings.) If you convert MTS. files they should also be AIC, if you are converting to H.264, these clips will need to be rendered when dropped into the timeline as FCE will not edit H.264 without rendering, if at all. I haven't tried this. I have used MPEGStream clip to convert from H.264 to AIC, edited, then exported back to H.264, that works pretty well, but I think you are starting with AVCHD.

August 9th, 2010, 12:12 PM	#10
Tom Cadwalader Regular Crew Join Date: Jun 2008 Location: Cheboygan, Michigan Posts: 79	Good Point Good Point, this can make a huge difference.

August 10th, 2010, 06:32 AM	#13
Tom Cadwalader Regular Crew Join Date: Jun 2008 Location: Cheboygan, Michigan Posts: 79	One thought Well it looks like the de-interlace for the entire clip is what is taking so long. FCE has to create 9 mins of video from your source material. The more movement the more work it has to do. I would offer 2 suggestions. 1) Do the de-interlace during the conversion to Quicktime and see what that looks like. 2) If you don't need a stand alone copy of the quicktime file, uncheck self contained. This will speed up the conversion from hours to minutes. I would test things with a shot segment of the movie. You could copy a <1 min segment with a lot of whatever you are trying to overcome, and drop that into a new sequence. This will let you try somethings, but not spend hours to see the results.

August 5th, 2010, 07:58 PM	#2
Michael Wisniewski Wrangler Join Date: Jul 2002 Location: Makati, Metro Manila Posts: 2,706 Images: 32	Try applying the de-interlace video filter to each clip and see if that speeds up the initial render.

August 5th, 2010, 09:16 PM	#3
Justin Beech New Boot Join Date: Aug 2010 Location: Sydney, Australia Posts: 9	I only have one clip in this example? it is a 10 minute long video, so start FCE with new project after log/transfer (or import 500mb MTS file - doesn't matter really), I drag it into the timeline, get a dialog saying something about FCE will change the format of the sequence to match, add the filter, then "render" and sit back and initially it predicts 5 hours, but takes maybe 3-4 depends what else is going on on the machine. It takes 150% cpu during this time. Then the bars above the time line are all red, and I am free to edit.. obviously it is impractical for me to import or log & transfer 100 MTS files and deal with each this way.

August 7th, 2010, 01:54 AM	#5
Justin Beech New Boot Join Date: Aug 2010 Location: Sydney, Australia Posts: 9	I'll try that checking settings, thanks. edit: what am I supposed to be looking at.. The only screen that seems to have applicable items is "Easy Setup.." there I can pick 'Format (all formats) Rate (all rates) and Use: AVCHD - AIC 1920x1080i60 ? thanks.

August 7th, 2010, 08:50 PM	#6
Tom Cadwalader Regular Crew Join Date: Jun 2008 Location: Cheboygan, Michigan Posts: 79	Yes this is where you set the sequence settings. They need to match what your camera was set for....to avoid a render. I also recall an issue with the audio, if stereo was not selected somewhere ( I can't recall where) there were issues. You said you recorded 5.1, FCE doesn't work with that and it needs to convert it to stereo somewhere in the process.

August 9th, 2010, 12:55 AM	#7
Justin Beech New Boot Join Date: Aug 2010 Location: Sydney, Australia Posts: 9	results: File from MTS via ClipWrap: 2.47gb, uncompressed audio (audio is the majority of the file). "H.264, 1920 x 1080, Millions AAC, 2 channels, 44100 Hz" (less than 9 minutes of footage) After setup for h-264, and New Project.. dragging ClipWrap 9 minute .mov file to FCE: instant Adding 'De-interlace' filter: instant Doing Render: Phase 1: Writing Video .. (Estimated Time: About 3 hours) (started at 10:18am) During this time, CPU usage is about 140% and disk activity comes in short bursts of about 500k/sec 11:35am - finished (1 hour 25 minutes) then .. Export as 720p and 5000kbit .. start at 11:50am, estimated time "about 45 minutes" During this time, CPU usage is about 150% Total time: 2h 30m including running ClipWrap first. vs ffmpeg, ffmpeg -i 00000.MTS -deinterlace -vcodec libx264 -vpre hq -s hd720 -g 30 -b 5000k -ac 2 -ar 48000 -ab 320k -threads 0 test.mov take native MTS file, converts to same .mov in 9 frames per second - 30 minutes. 180% cpu very small disk activity and low memory footprint. so does this sound about right? 2h 30m per 9 minutes of footage for Final Cut Express?

August 9th, 2010, 09:00 AM	#9
Guy McLoughlin Major Player Join Date: May 2007 Location: Toronto, Ontario, Canada Posts: 287	Make sure you are rendering to a different physical drive. Rending to the same drive that the AVCHD assets are on can take as much as 10x longer.

August 9th, 2010, 03:15 PM	#11
Justin Beech New Boot Join Date: Aug 2010 Location: Sydney, Australia Posts: 9	I'll try all hints in the last few posts, today, and see where that gets me. thanks. I do have 'scratch' drive set to an external drive, but yeah, I'll experiment some more. the de-interlace filter is necessary because the movie has fast motion .. nothing else gets rid of blurred fast-moving borders of high contrast that result from re-encoding these interlaced areas. We're not talking just a little comb effect on the edge of some things.

August 10th, 2010, 05:10 AM	#12
Justin Beech New Boot Join Date: Aug 2010 Location: Sydney, Australia Posts: 9	well, the results were not good. New Project, Drag .mov file into browser (9 minutes 1080i source material) Drag .mov file from Browser to otherwise empty timeline Add Filter .. de-interlace (red lines appear, but I do not "Render") Select Project again in Browser Export .. Quicktime Conversion .. (720p) .. the estimated time is 4 hours. Cancel that and remove the de-interlace filter Export .. Quicktime Conversion .. the estimated time is still 4 hours. Note, I'm writing to my home directory, and the FCE scratch disk is setup as an external drive.

August 11th, 2010, 01:48 AM	#14
Justin Beech New Boot Join Date: Aug 2010 Location: Sydney, Australia Posts: 9	I think I'm giving up on FCE for conversion of h-264 HD input .. Checking "de-interlace" in "quicktime conversion.." does not, and I didn't expect it to, because it also doesn't de-interlace when checked during exporting from Toast 9. On top of that, just doing the Export from FCE without any filters, just the quicktime conversion settings, STILL took several hours to run..

August 12th, 2010, 08:51 AM	#15
Tom Cadwalader Regular Crew Join Date: Jun 2008 Location: Cheboygan, Michigan Posts: 79	One last thought Justin, Maybe I should have asked this first, what camera are you shooting with?