Ok, so I just tried encoding part of the video two times with the same settings. The difference was that one was done without multiplexing and here the size of the ac3 and video file is 50% smaller than the muxed file!? So it seems that the muxing is adding a lot of size!
Under multiplexing there is only 1 option regarding "Mux rate" wich I have just left to 0. Should I set this to some value?

I solved my problem by not muxing the h.264 file in Adobe and just encoding seperate ac3 and m4v files. Now the filesize is right. I then mux with tsmuxer and to my delight discovered that this can also make AVCHD folders wich I then burn with IMGburn.

I tried it out in a stand-alone LG bluray player and it works nicely. Quality from the little Sony TG3e is great.