When you host, your connection needs to handle more traffic than compared to when you join another person's game. Thus, if there's other activities that's consuming bandwidth or if your router/modem has been on for a long period of time; sometimes this might cause the router to perform less efficiently.

So, if you have other downloads going on... then try stopping them.
Also, try to power cycle your router/modem (physically turning it off and on again).