Bd-5?
 

This is a very simple question:

How do I create a BD-5 in Vegas Pro 8? (BD on a DVD-R)

Tools - Burn Disc - Blu-ray Disc

Will the files be compatible with UDF and will it play on a PS3?

Edit: Tested it on a PS3, it will not play as a BDMV, but you can play the files in the "STREAM" folder as a AVCHD file if you rendered in Sony AVC.

Also, on Vista, it recognized it as a BD and you can access the file structure.