I'm wondering if I need to upgrade from Final Cut Pro 6. I recently got a 5D3, the footage looks good when I export to quicktime mov. but when I use compressor 3 it gets a jagged edge around all the edges and morie patterns appear, and I get a kinda blotchy look to the video, I don't see this in the quicktime mov.

Without going into the settings in Compressor because they are good (what Vimeo recommends) I'm wondering if the fault is in a version of Compressor that can't handle the 5D3.

I use the Canon plugin software to transcode the H264 to Pro-res 422.

Any thoughts about this?

I guess you should post something and point out what you are seeing and what you want us to comment on. Could be settings even though Vimeo recommends them. Since you have converted to pro res you should be able to find something that works even with your software. Some artifacts show up more on compression but also can be playback issues on the viewing device. Sometimes individual shots need to be adjusted for the output even if you don't see the issue at high quality.

FCPX eats everything you throw at it. I'm in a project that is mixing footage from a 5dmk3, T3i, HDV, MXF, GoPro and an iPhone - no problems. No more transcoding unless you want to (and you can carry on editing while the transcode works in the background).

Definitely time to upgrade; but do a course (like Ripple training) to get up to speed with FCPX as the workflows are a bit different.

Once you start using FCPX you'll be kicking yourself for not doing it earlier.

My old computers 2010 graphic card is failing so I bought a new Macbook 2.8 quad core i7 16 gigs of ram and a 764 SSD hard drive, and FCPX. So far I love it..... way better than FCP6!

I am getting very good results with Compressor 3 and DSLR footage.
I use the Log and Transfer to convert to Pro Res (though arguably you can get better results from 5DtoRGB, I find L&T an easier workflow), then export the edited sequence to QuickTime movie and open that in Compressor.
When transcoding for Internet in Compressor, I find increasing the bit rate can make a world of difference.
Here is the settings I use:

Description: QuickTime H.264 video with PCM audio at 48 kHz. Settings based off the source resolution and frame-rate.
File Extension: mov
Estimated file size: 6.44 GB/hour of source
Audio Encoder
AAC, Stereo (L R), 48.000 kHz
Video Encoder
Format: QT
Width: (100% of source)
Height: (100% of source)
Selected: 1920 x 1080
Pixel aspect ratio: Square
Crop: None
Padding: None
Frame rate: (100% of source)
Selected: 29.97
Frame Controls On:
Retiming: (Better) Motion Compensated
Resize Filter: Statistical Prediction
Deinterlace Filter: Best (Motion Compensated)
Adaptive Details: Off
Antialias: 0
Detail Level: 0
Field Output: Progressive
Codec Type: H.264
Multi-pass: On, frame reorder: On
Pixel depth: 24
Spatial quality: 75
Min. Spatial quality: 25
Key frame interval: 12
Temporal quality: 50
Min. temporal quality: 25
Average data rate: 15.36 (Mbps)
Maximum data rate: 15.36 (Mbps)

I just got FCPX a week and a half ago...... whipped together a little video to test out. I just hit share to Vimeo, and the footage looked way better than anything before.

Those were pretty close to the settings I used on Compressor 3.

I would say that compressor is pretty old at this point.
If you are using the internal codec, I would recommend 5DtoRGB for the Prores conversion.
In this way you can avoid the jaggies associated with the quicktime processing of most apple engines.
This is the reason for which that software was designed.