Brother, this problem has cropped up because of your own psychological assumptions. To build up your confidence, you need to be able to break all the limitations.

People say you are fat, you need to be brave enough to admit that you are proud to be fat, and if you're not, DO SOMETHING ABOUT IT! Don't sit down and complain. The more you act upon on these things that drag down your confidence, the more you become stronger in facing these challenges in future and makes you more confident.

Come join the gym and workout. The boost in energy levels and probably the improvement in physique will boost your confidence. If it doesn't help increase your confidence, I don't know what will...