I may be able to help with DD5.1 but not DTS. I use Vegas Video, and with it comes the AC3 encoder. Just set your project to 5.1, set your tracks to your discreet channels and encode away.

You can try DVD Architect for authoring. It should be a breeze after you learn the basics.

There was also a standalone AC3 encoder that I used, the Sonic Foundry Soft Encode. That was old, like 7 years or so, but it worked great.

As long as you can open the file in vegas & set which tracks go to which channel, then yes.