My 2 Sen:
Go see a good marriage counselor together. Unfortunately some gave you poor advice. From what has been said, there are multiple issues here. It won't be easy but should be workable and definitely can improve. I hv worked with various couples on their marriages, young n old, rich n poor. Basic understanding on what marriage is varies from one person to another and that must first be sorted. Then there are the other aspects that needs to be sorted and will definitely improve/enhance yr marriage/family.
I wish you the best - you n your family.

Not true bro. If you experienced this, maybe you can get another one. Takes 2 hands to clap. Both parties need to make the marriage work.