Big Big Problem with my 1709!

I didn't have this problem with getting Dolby Digital or DTS through optical connection from either my DVD player or Asus O!play. However, I've given it to my brother and he tries to do the same with his setup but he is ending up getting loud noises from his speakers, which are all custom made ranging from 6 to 16 ohms, supposedly supported by the amp as par the manual. I've looked in to the set up and couldn't figure out the problem. I've changed the setup at the O!Play to LPCM then the sound is clear for Dolby Digital but no sound for DTS. Further when I change things to raw the same noise persist. And after a couple of minutes of the loud noises the amp automatically shuts down the sound. I've tried changing the cable, the input (i.e.
input 1 & input 2), the DVD player, the speakers - the problem still remains.

I don't thing there anything damaged in the amp but I still couldn't figure out the problem. I've run the Audessy setup which works fine, just like any playing other sound source from analog. What I want to know from you people if you know I've done something wrong with the setup which I can rectify or there is a reset button which can make it to the factory default setting.

I live outside the range of any Denon service center, so I'm depending on you guys help.

Thank you all for your time.

P.S. This is already posted in the Denon Amp forum. This is reposted for further responses from other contributor to this forum.