I'm only guessing here, but it's probably because of how the first aircraft control surfaces were wired up. Pulling up the stick causes a wire to pull the flaps down, and cause the aircraft to go up. After a while, this become convention, and everyone follows it.

But what do you mean the controls are inverted. It isn't. You pull up to go up, you push down to go down.

Would you prefer that you pull up to go down, and push down to go up? Would you also prefer that you turn right to go left, and turn left to go right?