Marriage is about husband-wife having the common goals and finding the balance between your differences. In your case, sit down with your husband to get the priority right. This encompass your physical, mental, spiritual and emotional need. If this cannot be done, I would say this is end of the road.

It's pity your baby didn't receive the love. This is the golden moment which I think you need to take charge despite your daily heavy workload. Putting your baby in the hands of others is like a time bomb. You should should to your husband that having baby means responsibilities. If he can take care of the first one, it fine to have the 2nd one then. Else ask him to forget about it.

Hopes it helps.