HDV: How much can I compress?
 

Here's a question: I've got 28 hours of HD footage shot in 24p. After it goes through reverse telecine, each hour of footage takes up about, oh, 30GB or so. I have 28 tapes, and while I have a 1TB drive, I'd rather set it up as 2 500GB RAID drives for backup.

Would it be possible to compress all the video to H.264/best and edit using that? What kind of quality loss am I looking at? Can I save space WITHOUT losing quality by inverse telecining to Apple Intermediate Codec instead of Photo-JPEG?

What would you recommend?

28 hours on a 1GB drive! Now that's what I call compression!!!! :=)

Three things;
1. 2 - 500GB drives set up as Raid 0 still equals 1000GB (1 TB) of storage.
2. Do a web search. There are several free widgets and programs that help you compute storage space vs. codec.
3. ProRes422 uses up approximately 115 GB/hr. Neither Photo -JPEG or AIC will look as good as ProRes.

You can either apply heavier compression (and lose picture quality) or buy more storage.

Each time you compress and then un-compress and compress again you loose quality with video footage... its not like compressing a text file...

compression is not a good work flow for archiving video footage...

Should be 1TB.

Avoid recompression. How come you need 28 hours of footage?

he said raid for backup. a raid for backup is a raid 1, and would write identical info on both drives

Making a documentary, with a shooting-to-film ratio of about 20 to 1.

I'm confused. Your title mentions HDV but you have 30GB/hour HD footage from reverse telecine. That means it can not be HDV. What type of tapes do you have? What is the encoding format? What will be the final output format?

I'm recording to DV tape in HDV format 1080/24p via the Canon HV20.

The files that come off the DV tape are much smaller than the files that result from the reverse telecine in photo-Jpeg.

The final formats will be:

Theatrical Distribution (maybe), H.264 1080/24p Blu-Ray, DVD NTSC, DVD PAL, and possibly H.264 1080/24p Online through iTunes.

My big concern is that I want to get a copy of this footage over to my movie-making partner in New Zealand. What I'm probably going to have to do is buy another 1TB drive and just ship it straight to her with the files on board.

There's another factor. Alot of this footage is over multiple tapes, so what I'm doing is I'm putting them together into a full interview - about 30-60 minutes long, and saving as uncompressed Quicktime.

Maybe I should save those files as MPEG-2? I mean, HDV is already in MPEG2, so...

I still don't understand something. You are doing a reverse telecine "filmout" BEFORE you edit? Why are you converting to photo-jpeg? It doesn't make sense to me to convert to a larger format then re-compress for editing. What exactly are you trying to accomplish?

It seems to me that you should have edited in HDV or an intermediate format like the Cineform codec. After all the editing is done seems like a more sensible time to convert to a format for reverse telecine. If you edit in HDV, you can simply mail a copy of the tapes to your partner. It is also possible to edit in a low-res intermediate then substitute the HDV files later with an edit decision list but you need to make sure your timecode is all good or the edits won't sync. I have not used that process so somebody else will need to explain that process.

Again, the big issue I see is that you are doing a conversion to a reverse telecine format before you have even edited the project. Have you consulted the house that will be doing the conversion to film so you know exactly how they want it done?

I use the Canon HV20. It's a consumer-level camera that records HDV in 24p. However, it records it in a pseudo 60i format on the HDV tape - AND it doesn't put in the required flags to mark the interlaced frames.

To get around this, I have to download the material as that "pseudo 60i" format, and use a program called JES Deinterlacer which will detect the cadence breaks and perform the inverse telecine to make it a normal 1080/24p .mov file. This file uses Photo-Jpeg, although I could use QuickTime to compress it out of the box.


If I had edited the movie first, then used reverse telecine, it would have messed up any transitions, possibly messed up color corrections, and may, indeed, not have worked at all.

In short, my raw footage is therefore in Photo-JPEG.

Unfortunately, if I record the Photo-JPEG files back to HDV tape, it will turn back into that psuedo 60i format. Otherwise that would have been my first choice.

I'm not converting it to film.

---------------------------------------------------------------

This may sound like a bad idea to many, but I've decided that I'm going to convert all the files to H.264 1080/24p. Yes, I'm adding compression early in the process, but I've thought about it and it just simply makes sense.

The H.264 compression is just as good as if I had recorded in AVCHD rather than HDV - better actually because I'm using a higher bitrate than AVCHD. I'm recording on consumer level equipment to begin with. Yes, the HV20 is an awesome camera but it's not the same level as the Panasonic HVX, and the quality loss is minimal and the speed at which I'll be able to work is greatly increased by having files that I can send over to New Zealand by burning them onto DVDs and mailing them. It also means more room on my scratch disc, that I can co-locate them (It's easier and less expensive to find a hosting provider for 200GB of material rather than 1TB) to prevent data loss.

Additionally, I really, absolutely positively, cannot tell the difference in quality even when going over the pixels with a fine-tooth comb. Whatever compression artifacts H.264 introduces, it's not as bad as the MPEG-2 compression already on the tape.

I'm glad you cleared that up. The 24p workflow is a bit mysterious.

I assume you are doing this?:

"This file uses Photo-Jpeg, although I could use QuickTime to compress it out of the box."

Are you able to select a different compression in JES Deinterlacer than the huge format you had before or are you just running a re-compression on the files in Quicktime? It is probably best to have the fewest operations performed on the data but if you can't tell a difference it should be okay as long as you keep other compression generations to a minimum.

Indeed, I'm choosing Photo-JPEG out of JES Deinterlacer because that's an uncompressed format. I then take the uncompressed footage, bring it into FCP, string the disperate footage together into one single file, and color correct. Then I export that as an uncompressed .MOV file, and use Compressor to convert that .MOV file to h.264.

The real quandary is if I can delete those uncompressed .mov files (around 30 GB each) to save space, because the H.264 data (around 5 GB each) contains the same information.

Since you are doing color correction already while the file is un-compressed or losslessly compressed, it seems that you no longer need the original Photo-JPEG files as they have fulfilled their goal of converting to 24p. Now, most of what is left to do is cutting and audio so you shouldn't need any uncompressed files. It's not disastrous to do color correction and the like on compressed files anyway.

I am curious as to one thing. Are you getting 5GB/hour with your h.264 compression? That is highly compressed and may be challenging to edit. If you aren't having sluggish performance then you shouldn't worry if it is going to be the same as your highest quality distribution format. If you are simply going to add cuts, fades then tweak the audio, this same H.264 format can be your HD master. Assuming that your only conversion from there will be to a lower SD format, you should see no significant further degradation. I don't know how efficient the conversion from H.264 to MPEG2-DVD will be, but you are at least maintaining a greater-to-lesser conversion path.

I should add that HDV is about 13GB/hour but it is supposedly less efficient than H.264 compression. You may now be editing at lesser than HDV quality, but you have already done the step that really needs the quality (color correction). I don't know if you would get better results going to DVD if you converted to SD from a greater format, but it is something you might want to experiment with before trashing your big files.

Indeed, I'm keeping the un-color corrected files around in 24p as they came off the tape as my "backup" (plus, I also have the tapes!) Editing from the H.264 stream means a quicker turnaround time and it allows me to store a copy of my raw footage on my online account as well as transfer it over the Internet to partners around the world.

I would expect around maybe 7GB/hr for an action packed video, but most of these videos are of interviews - hour-long interviews. Background doesn't change and the talking heads don't move too much. I'm not surprised at the 5GB/Hr video rate.