Yes, the RCA outs are analog.

Hmmm. Whenever I calibrate the monitor on set, or anywhere, I've always had it hooked up via the S-Video port on the camera, never the RCA. Is S-Video considered digital or analog?

The S-Video output is analog, not DV or digital. But technically it is not NTSC RS170a. RS170a specifies composite video not separate y/c. They should be pretty close, but you never know.

A couple of things that are easy to overlook when connecting a camera etc. to a scope (not software scope). The scope may be set to internal or external sync and not syncing to the camera or other source. The signal may be terminated incorrectly (either double terminated or not terminated at all). Improper sync or termination could be causing your visual problems. It doesn't mean the signal is incorrect, just the scope isn't set-up right.

Okay. . .if I go back and try again, I'll keep all that in mind.