Best HDV workflow.
 

I need to know if there is a better workflow out there. I'm currently capturing HDV as ProRes422 and passing it via external drive to my editors. I usually give them 3 weddings per drive and that fills up the drive. I also keep a copy on my Mac Pro. I have 3 editors working at all times, so thats a lot of weddings to keep stored. I also keep the original files on the computer for 4-5 weeks after delivery to the bride, just in case there are any problems.
My job seems to have changed to data wrangler at this point. I've been told editing in HDV is slower than ProRes422, but this file size is killing me. Any thoughts?

Great question.But an incomplete question perhaps?
- what "external drive" are you using for capture?
- Please review this work-flow diagram. Ca you tell us what the rest of your turnkey system looks like?
http://img.photobucket.com/albums/v2...20WorkFlow.jpg

I shoot HDV (Canon 24F mode with XH A1), capture in the HDV 1080P24 setting, edit in a 23.98 HDV timeline, export to whatever I need using Compressor. FCP handles HDV perfectly well, so I don't see any need to do it any differently.

You should consider capturing in HDV and having your editors set their render settings to ProRes. This way you save drive space (and transfer time) and maintain quality in the one place where HDV falls apart in quality and time, renders.

That is a good question.

I know when editing HDV, the quality can begin to get a bit unglued. If you use HDV for your source, but render out to Prores or cineform (plugins), will the original HDV file be affected? I know this might sound a bit silly, but it is something I've never read an answer to.

Thank you.

But I thought the whole point of ProRes was to get it captured to this codec to edit in to preserve quality while adding effects, color correction or whatever.

Simply rendering to ProRes (from HDV) after all the editing seems to me would be missing many of the advantages of the ProRes format.

And if you capture to ProRes to begin with, at least you only have those files to worry about begin with.

Jeff,
When working with HDV, the compression is done in the camera before the data is written to tape or outputted via firewire. The loss due to HDV has already occurred. When you transfer HDV to your NLE by firewire, there is no additional loss (just like copying a file bit-for-bit).

Pure HDV workflow:
When you put these HDV files on your HDV timeline with only cuts, there is still no loss because you aren't recompressing the data. The only time you get loss is when you have to make changes to the video, such as when using filters, titles, transitions, etc. If your timeline is set to render to the HDV codec, then any parts of your video that are changed will be decompressed, have the changes applied, and then recompressed in HDV (but other parts which are unchanged will remain in their full original quality). It is this recompression step that causes loss, since HDV is a pretty lossy codec. It also makes editing slow, because decoding and recompressing HDV is a lot of work for your CPU to do. The advantage of this workflow is that it uses the least disk space, and unchanged parts of the video incur zero quality loss. So this is a good workflow to use when doing mostly cuts-only editing with minimal rendering needed.

Mixed HDV/ProRes workflow:
If you use a HDV timeline but with the render codec set to ProRes, then this is what happens. Parts of the video which are unchanged will stay in HDV as they are without any recompression, so there is no loss. Parts that are changed will be rendered in ProRes instead of in HDV, which minimizes the loss. This workflow gives you most of the advantages of the pure ProRes workflow, but uses much less disk space (since your source files stay in HDV which takes much less space than ProRes) and doesn't lower the quality of unchanged parts of your video. The downside of this workflow (compared to pure ProRes) is that, since HDV is harder for your CPU to decode in realtime, you get slightly fewer realtime effects before needing to render. The upsides are that rendering is fast because ProRes is relatively easy to compress, you get minimal quality loss in changed parts of your timeline (and no loss in the unchanged parts), and the disk space required is only slightly higher than with the pure HDV workflow.

Pure ProRes:
The pure ProRes workflow involves transcoding the HDV data that comes over firewire into the ProRes codec, so the files on your disk are all in ProRes instead of HDV. Since ProRes is a (slightly) lossy codec, this means ALL of your video will LOSE some quality (although the loss will be pretty small). While it is true that ProRes is a higher quality codec than HDV, transcoding HDV into ProRes doesn't increase the quality, it only gives you the compression artifacts of HDV PLUS the artifacts of ProRes. This also requires a great deal more disk space, because ProRes uses something like 4-8x the bitrate of HDV. The advantage of this workflow is that since ProRes is easy for the CPU to decode and encode in realtime, you can have more realtime effects and faster rendering than with a pure HDV or mixed HDV/ProRes workflow. The downsides are that all of your video loses a little bit of quality and much more disk space is required.

So, in summary...

Pure HDV Workflow
Pros:
Least disk space
Zero quality loss in unchanged parts
Cons:
Noticeable quality loss in changed/rendered parts
Lowest realtime performance (both playback and renders)

Mixed HDV/ProRes Workflow
Pros:
Uses only slightly more disk space than pure HDV
Zero quality loss in unchanged parts
Minimal loss in changed/rendered parts
Cons:
Less realtime performance than pure ProRes (but faster renders than pure HDV)

Pure ProRes Workflow:
Pros:
Maximum realtime performance
Minimal quality loss in changed/rendered parts
Cons:
Slight quality loss in unchanged parts
Uses a lot more disk space

Thank you Jason. Couldn't have explained it even close to the way you did.

How would you go about creating a ProRes and HDV?

Thanks Jason, crystal clear!

Robert

Following up on Jason's lucid comparison, the questions that an editor might ask themselves are:

- How noticeable to the customer is the degradation of quality in areas of the edit that have been filtered/transitioned? I guess three clips from the three different options would be great to review.
- Is the additional disk space that's required for a totally ProRes edit paradigm always worth the extra expense or extra physical drive arrays?
- Seems like the pure HDV and mixed capture/edit paradigm advantage laptop-based edit turnkeys?

The render degradation is very noticeable if you pause the timeline in a pure HDV timeline. A client isn't always the best reason to skimp. Sometimes you want something good to show.

The additional disk space is needed if your project requires it. Put the cost of the drives in the project. A multicam switch is better in full ProRes. A standard edit job works great with the HDV/ProRes render setup.

If I understand you then yes, HDV works great on laptops for standard editing jobs.

Dont you need a card like decklink or kona to capture prores it cant be done via firewire is this correct?

Incorrect, the capture from FW is the HDV signal that gets printed to Prores format if you choose to use it.

It is like transfering one format to another format, you are not capturing Prores, but transfering to it while capturing.

I just read your post Jason, as this forum was not sending emails when posts are followed up on.

But you in your wonderful explaination of pros and cons of Prores/HDV codecs did not address the issue I brought up. We are talking about editing, and not what are the shortcomings in HDV at the camera stage.

And that is that rendering out to Prores after editing/composing/transitions, etc. defeats the purpose of Prores. My posts stated that you must get in Prores before doing these things to take advantage of the codec, and not simply render out to prores.

You did not address this issue. But nobody seemed to notice anyway.

It depends on the customer, but I found out that the degradation is heartbreakingly bad. Like the difference of going from HDV to SD. You are knocked down a flight of stairs in quality.

Yes, it is worth the cost of a cheap drive to capture into Prores and work that way.

I did only simple wipes and desolves, a few fade ins and fade outs, and it ruined my HDV footage. I will do that entire edit over again.

Ok So i will capture Prores HQ which i already tried through my canon xh-a1 data rate was around 18mb/sec and capturing in HDV data rate was at 3mb/sec

So i guess capture in Proress ,edit in Prores and export i guess in what ever format i need or in pro res

Yes, this is the way to go like a "Pro". Some folks (posts in this forum) will export to Prores, and then take that finished file and drop it into a SD timeline to create a new SD file. That is just one way to create a SD file out of many.

But bottom line is that capturing it originally in Prores is getting you off to the correct start. Drives are cheap, so no reason to delay your workflow.

re
 

yes i also think pro res is the way to go

i have spent the last 5 months trying and testing every combination and the best results are, or in my opinion, capture pro res 422, edit pro res 422 ,1480 x1280 25p, export apple prores 1480x1280 25p, drop than into compressor,tweak up the dials, best quality SD DVD 90 mins , then into dvd studio for output.

The quality is fantastic. Almost looks HD on my 42inch, so sharp.

Also isnt pro res better with color, im sure i read that but.........


luke

just need some terabyte drives now

Jason,

That was a great reply.
It is the workflow I have been using for a year and a half
and it works very well.

David

Depends on the model of Mac you have. The Dual 2g G5 I have can capture 720p30 to ProRes directly thru FirwWire but not 1080i. Final Cut seems to able to assess your computer and will give you the option.

Your understanding of the process needs a little sorting out. The shortcomings of HDV are after the camera, it's in the recording process and the editing process.

When you edit in HDV you are cutting the unadulterated video files direct from tape or camera. That is an m2t file (GOP-based MPEG2 transport stream) that has a QuickTime wrapper placed around it to enable frame by frame editing. The m2t file is untouched at this point if you capture it via FCP or use ClipWrap on the raw m2t file. No quality is lost or gained here. A cuts only edit in HDV is as good as HDV will ever get. Once you apply a transition or filter, the m2t file has to be re-encoded back to a m2t file. As we all know, this doesn't lend itself to clean renders and the process requires a lot of system resources.

ProRes is a frame-based codec not a GOP codec like HDV so it requires less system resources even though the file sizes are larger and it works in a larger color space. By transcoding HDV into ProRes you get all the benefits of a frame based coded and the fact that ProRes employs less compression than HDV therefore all renders will come out cleaner. But....

Transcoding is not a transparent process. You will loose a very minor amount of quality but the new files keep the resulting quality high during rendering. Also you are rendering in a better color space than in HDV. That helps a lot.

By setting your timeline to HDV with ProRes renders you keep the quality of your cuts only footage and the renders are high quality without using tons of disk space. FCP plays the footage seamlessly. This can be outputted to HD tape via a number of methods or rendered via Compressor for DVD or to Uncompressed files.

This process does not defeat the purpose of ProRes, it takes advantage of the very qualities that make ProRes an attractive codec. If you are working with specific input/output hardware that uses ProRes, then stick with a full ProRes work flow otherwise you are wasting drive space in a situation that you have described as drive space sensitive.

Try a test edit with this method. If it doesn't work for you then it doesn't. It works for a lot of other people.

I'll expand a little on my original post and what William (correctly) stated.

For the quality comparison numbers, let's say 10 is the max (pure uncompressed quality), each pass of HDV reduces the quality by 4, and each pass of ProRes reduces the quality by 1. (These have no connection to any real-world metric and are for demonstrative purposes only).

--- Pure HDV ---
In the pure HDV workflow, the files captured by firewire have 0 quality loss (compared to the compressed HDV stream recorded by the camera), it is just like making a copy of a file. When you play back HDV on your timeline, you still have 0 quality loss. When you need to render a change, this is what happens. First each frame is decompressed to an uncompressed frame in memory (neither losing nor gaining any quality). The filter/title/etc. functions are performed, and then the modified frame is re-compressed into the HDV codec (causing a large loss in quality due to the limitations of the HDV codec). But frames which are unchanged don't go through this process. They retain exactly the same quality that the camera recorded with 0 loss.

Pure HDV Summary:
Unchanged parts: HDV artifacts (Quality: 6)
Changed parts: HDV artifacts + more HDV artifacts (Quality: 2)


--- Pure ProRes ---
In the pure ProRes workflow, as you are capturing the video over firewire, each HDV frame is decompressed into an uncompressed frame buffer (neither losing nor gaining quality), and then compressed into the ProRes codec (causing a slight quality loss, because ProRes is a lossy codec). That is to say, every single frame of your video captured in this way is going to lose some quality. Not a huge amount, but it can mathematically be proven that some data WILL be lost. This is the nature of any lossy codec, which is what ProRes is. When you apply effects/filters, each frame is decompressed from ProRes, has the effects applied, and then recompressed into ProRes (causing a further slight loss).

Pure ProRes Summary:
Unchanged parts: HDV artifacts + ProRes artifacts (Quality: 5)
Changed parts: HDV artifacts + ProRes artifacts + more ProRes artifacts (Quality: 4)


--- Mixed HDV/ProRes ---
In the mixed HDV/ProRes workflow, frames which are unchanged will retain their original data and lose no quality. Frames which need to be changed will be decompressed from HDV into an uncompressed buffer in memory, have the effects applied, and then re-compressed into ProRes (causing a slight loss, but much less than when compressing HDV).

Unchanged parts: HDV artifacts (Quality: 6)
Changed parts: HDV artifacts + ProRes artifacts (Quality: 5)


--- Bottom Line ---

The mixed HDV/ProRes workflow gives you the best possible quality (outside of resorting to uncompressed), with only a minor decrease in real-time editing speed and minor increase in disk usage.

The pure ProRes workflow gives you an increase in real-time editing speed, at the cost of slightly decreased quality and greatly increased disk usage.

The pure HDV workflow has the least disk usage but the worst quality (unless doing cuts-only editing) and the worst real-time performance.

The ONLY advantage to the pure ProRes workflow is if you absolutely need the maximum possible number of real-time effects that can be played without rendering. Any sections that need to be rendered will have a slight quality LOSS compared to the mixed HDV/ProRes workflow. Any sections that don't need to be changed will also have a slight quality LOSS compared to either the HDV or mixed HDV/ProRes workflows.

Of course there is loss that has already occurred as HDV is a mp2 compressed file. But there is about to be more loss during editing with any transitions, etc., unless you get it into another codec quickly.

I never said that there was loss by transferring the files bit by bit, I said nothing of the sort.

The recording process is in the camera just as Jason says above. The compression happens there also.

With these long detailed posts, there still is not the suggestion that you must get into Prores right away to avoid the drop in picture quality. That was my statement, and it remains my workflow. I don't know any editors that do only cut, insert, cut, insert - either. All my projects require something more than just a simple cut.

Professionals who work with HDV always get into the codec that they plan in working in at the earliest stage of the project. That is my point and I am sticking to it. We can go around and around about what degrades the picture quality worse (cuts or transitions), where Prores' qualities are most apparent, and how well it works with color corrections, etc.

The bottom line is that you want to get into your working codec early as possible (which ever one it will be), this has been taught to me from working professionals, and I believe them. All this talk about where the compression (and loss) happens in HDV is a side issue concerning cameras and the way they record, and if it happens in or after camera is of lesser concern. Jason says in, William says after.

The beginners who are asking about Prores workflow need to know to get into it at capture if they are at all able to, and they plan on doing any serious editing. That is the important issue concerning workflow.

Sorry, you are misunderstanding the process still.

A camera is a camera. It has it's qualities, good or bad. Back in the day of separate camera / VTR packages, you could take a great $50,000 camera (intended for BetaSp, 1 inch) and hook it up to a portable VHS and get the best VHS recording possible but it still had all the problems of VHS. These days a camera is usually wedded to a specific recording technology, DVCam, HDV, XDCam, whatever. A good 3-chip HDV camera might be making better images than the format can show but unless you are willing to go thru some difficult rigging, HDV is what this camera is going to shoot. Maybe there will be a ProRes recording camera but now there isn't.

Rendering issues have nothing, please take note, nothing to do with the camera. It's all about the recording codec.

Maintaining quality from HDV footage is what Jason and I are talking about. There are many work methods, you asked and we suggested. Please don't be offended if we did not concur with what some other professionals told you. They are not wrong and neither are we. Are you at least going to try the test?

In the end, if going straight to ProRes fits your editing style and works with your other editors, great. But it's not the only way to work and please do suggest that we are misleading beginners because we are not. The methods put forth by Jason and expounded upon by me are serious editing workflows, used by professionals. Us.

Then why are you going on about cameras? And your tone is insulting.

Thanks and Recovering Losses in Quality via Re-rendering
 

Jason & William,

Many thanks for shedding light on what, for me, is still a somewhat mysterious process of achieving best results (i.e. min loss of quality/res) while editing HDV. Jason, you ought to be a teacher with your gift of clarity!

I have one question for both of you and forgive my ignorance in advance. Here’s my scenario and then the question will follow: I shot a multicam wedding last year using three JVC HD100 cameras and captured my 720-30p footage into FCP2 as per standard methods. However, I completed most of the editing last month BEFORE I was enlightened by your recommended HDV workflow, adding my transitions and filters along with multi-camera overlays/masks/feathering, etc, while leaving the “Render control” sequence settings to “Same as Sequence Codec” (i.e. I never knew the advantages of switching them to ProRes before I started editing my project).

My question is, now that I have already added transitions, etc. CAN I “RECOVER” THE LOSSES IN QUALITY caused by allowing FCP to render in HDV if I go back and change the sequence render settings to ProRes and then re-render the project, and so on?

If not, (sorry, a second question), how might I go about re-doing my edits without starting from scratch? (i.e. do I simply remove all transitions and/or re-import files and footage onto my timeline and/or cut and paste everything into a new sequence set to render in ProRes, and/or…??)

Thanks so much in advance for any further illumination.

Michael Lafleur

The way to minimize loss is to keep the unchanged HDV segments in HDV so that they have zero loss on the timeline compared to the original. Changing them into another codec, no matter how quickly, will only cause unnecessary loss (unless you're talking lossless or uncompressed codecs, which ProRes is not). For rendered/changed segments, you want to minimize the number of lossy codec cycles the video has to go through. Using an all-ProRes workflow adds an additional lossy cycle which in most cases is unnecessary.

I think it's unfair to imply that "[all] professionals" "always" use the workflow that you prefer. I happen to know of many professionals that would disagree. There certainly are some cases for which the all-ProRes workflow is best (if real-time performance or rendering time is the most important concern), but in doing so you are sacrificing quality and disk space.

I think that beginners should choose the workflow that works best for them and meets their needs given the pros and cons of each that I outlined in my post above, instead of blindly sticking to a single workflow based on mistaken or incomplete information.

Yes, you may need to trash your render files to force FCP to re-render, but that's all. Your original source media remains untouched in the capture files, so by re-rendering all of the effects and filters will be re-calculated from the pristine sources and you'll generate new, high-quality renders. Any un-changed parts of your timeline will stay in their pristine source format(s) and not suffer any generation or rendering loss at all.

Great News…
 

Thank you for such a prompt and helpful reply! I have never tried to purge render files on a project but I assume they are easy to locate on my scratch disk and the process is quite straightforward. If I need some assistance, I will ask for some basic coaching once again.

Kudos to You from the Frozen North

I think you are pointing out a benefit of not using Prores on all clips, but just the ones that will be affected by rerendering due to what ever effect applied. I'm deep into some kind of effect on everything I edit, being as simple as a cross-desolve, wireframe/size reset, or color correction, it's pretty bad.

I hear it is minimal loss (with Prores) comparatively speaking, compared to rerendering.

I agree that there is good reason to avoid the number of cycles the video (HDV) should go through. But I can't get much good work done with simple cuts and joins as my first comment in this post mentions. In "most cases" one will need to do some real work with the footage. It does indeed depend on the editor and what they hope to accomplish. The more advanced, the more need for Prores.

I am glad to hear others comments that Prores adds an "additional" lossy cycle. It makes sense, but I consider it the first and only rather than an additional. The HDV stream is what it is, and there is no getting around that we all know, it still is better than SD.

So i capture in the HDV codec and change my settings within my sequence to prores and edit that way

Do you think the decrease in qualty is noticable to the point it looks like SD

Because i shoot weddings i need the fastest possible render and realtime preview. I like going back and checking my work as i go along. I Use alot of slow-mo and speeding up the clips and colour correction weather it's natress effects or colour grading with the 3 way wheel. I would have to say Using prorez from the beginning to end will probably suite me best.

"cuts only"? can't recall the last time I had something like that!
 

In the "HDV+ProRes render" paradigm, how does colour correction come in to play? I can't tell you when the last time I had a project that had any part that was "cuts only". At the very least there is always some colour correction on each and every clip.

Now that I am working with FCS2 and have Color, I am using that instead of the 3-Way CC filter which will render out to ProRes anyway. Maybe I am answering my own question here... I suppose the workflow would be to do the editing on an HDV timeline, then send to Color and have my ProRes output as a result.

But let me throw this into the mix: what if you are doing some retiming/deinterlacing? Would AIC be a suitable alternative anywhere (for example when pulling 25p footage down to 24p for editing at 24p) or is ProRes still the way to go?

I can't imagine that would be the case. I have yet to actually work with any HDV. I am just about to start playing with some test shots Vito was kind enough to shoot for me and I will be looking at this stuff on a nice 24" HD broadcast monitor and really compare to see what the real deal is. Is the loss in quality even perceptible to the human eye? I don't know... I'll be looking for it...

Yes it is even worse than SD. The decline in quality of HDV footage when rerendered is heartbreaking and depressing to the degree that you were originally proud of your footage.

It comes out looking like a DVD encoded for the longest running times, like a 2 hour single layer DVD I kid you not.

I'm a relative noobe to HDV but after trying all kinds of solutions to get my HDV timelines to output decent quality (mostly going to sd DVD) I tried ProRes 422 HQ today (QT converted an edited HDV timeline) and the results were a big improvement. I'm now trying one captured in ProRes.

I used Ken Stones guidelines. There's also an older HDV to SD DVD one on his site that is pretty basic and does not use Pro Res. It works, but the quality is not as good.

Exporting HDV Video from the Timeline to Standard Definition DVD

My apologies.

Transcoding to ProRes drops the quality a very small notch but it's still HD and looks it. Most people will not see a problem.

An example of whether or not to capture in ProRes or stay in HDV. I do several multi-cam concert events a year which I switch in the edit room. I try to use the same model of camera to minimize color-correction issues in the edit. One concert I captured in HDV but edited with the HDV/ProRes render sequence set-up described earlier. After all was finished, about 3/4 of the sequence was rendered due to dissolves and color correction. That ends up being about 1 hour of ProRes size files. The last concert I did, I was worried that the cameras were not balanced properly at the shoot (I did the balancing) so I captured entirely in ProRes. I had to buy an extra drive for the five hours of ProRes size footage. It turned out that I could have stayed in HDV, only one camera needed some work, but.....

It turns out that FCP puts out a real time preview of ProRes sequences over FireWire in standard def which you can send to a standard def monitor. It doesn't look great but it's good enough for the client and for color correction. You can also get a real-time preview of HDV if you edit it in a ProRes sequence but the quality is lower.

Did the concert look any worse for being edited in ProRes? I don't think so.

My Canon XH-A1 shoots 1440 x 1080 when i export from the timeline as a quicktime file it comes up as 1920 x 1080 WHY?

I was showing a friend that uses edius how compressor works and the time it takes to encode HDV it took 2 and a times the normal length of the clip but he insists i am doing something wrong because it took him 15hours to encode a 2hour clip.

I haven't burnt anything on DVD to play it so i can see but from what i can notice i am not doing anything wrong apple is just a dominant species

re
 

robert have u tried looking at the setting that you are exporting at. click the drop down where it says same as sequence and choose 1440x1280

see if that works??