Yes, you would need a math degree!!

And yes, I think it would be more difficult ... if you're not excellent in it, you won't survive. Now you're learning the simplest form of maths, if you can't survive now, you won't be able to survive when you learn about more abstract, something in a higher dimensions, and you'll learn new symbols and notations that can be completely different from before. In higher level maths, your calculator won't be able to help you.

Go engineering instead if your math is just so-so... and in engineering you've more option later, you can choose to specialise after 1st or 2nd year.

Does your math teacher has a math degree, or just a general science degree??

If you don't understand further maths (i.e. only know the steps to solve a problem without understanding, then a maths degree is not for you). I'm not sure about the degree offered in Malaysia, but overseas, maths and natural sciences degree are very rigorous and demanding. Having an interest and excel in rudimentary math are both necessary requirements, but not a sufficient requirement in order to succeed in this field. You need to have a good math intuition, logical thinking skills and critical analysis.

3 subjects at A-level are more than enough. Most angmo only take 3 A-level subjects. It doesn't matter what engineering you choose, most engineering degree starts with the same basic in the first 2 years, they only specialise and decide in 3 years. Check with your respective uni.

During my studies in UK, I've seen a number of students changing from a math degree to engineering, i.e. they start from year 1 again after spending 1 year in the math degree line. The same with physics. Haven't come across anyone that changes from engineering to math/physics.

An engineering degree is more applied, whereas a math degree can be quite theoretical and abstract. You can of course opt for applied maths in the later years, but first 2 years you need to learn quite a wide range of maths, which is not covered in A-level, so it's hard to explain to you.