Are you running any torrent clients in the network? What kind of router/modem do you have there? Does this problem persist if you're connecting direct from your PC to modem?

Reason being, when using torrent on some of the routers, too many connections done by the torrent client will hit hard on any router. One way to solve this problem is to put a limit in your torrent client.

If the above don't apply to you, check the values returned by your modem (SNR margin & line attenuation) if they're changing (SNR dropped really bad).. if yes, then probably your line wouldn't stand the 4Mbps and probably need to downgrade.