Downsides of refinancing:

1.) Penalty
Most of the banks are offering zero entry cost means you could switch to other bank with F.O.C! But still there might be some penalty inccurred: 3% for early settlement within lock-in period (normally 3-5 years) from previous bank.

2.) Again, to be locked for another 3/5 years
For those people who buy property for investment purpose, if you refinance your property, you can't sell within the lock-in period.

However, the above disadvantage can be counter off:

1.)Penalty
what if the interest saved when you refinance can be easily off set the penalty?
E.g : existing loan in bank A since 2004, initial loan 120k and current outstanding 100k, penalty 3%, lock-in period 5 years.

Put in this way:
Existing
installment: RM700
total penalty 3.6k
remaining lock-in period : 2 years.

After refinancing
installment: RM 500
interest saved during the 'remaining' 2 years comparing to prevous loan : RM 200 * 24 months = RM 4,800
In this case, the interest saved could easily off set the penalty.
2.)investment property
I believe when you're selling off your property, you must have a minimun price in mind. just take the penalty into consideration and add up to a total price that you want to sell.