Don't be so naive laa.. Yes they can't withdraw using just your account number. But it can be use for verification purposes.

Somebody call you.. Saying.. "Are you Mr so and so, account number bla, bla bla ".. Right away your brain think that this somebody is genuine right..
(Because they already have for private info)

So after that you'll bogel everything.. That's how it works.. Like Macau scam.. How can they trick people to transfer money to them so easily..
They must already have some P&C information of the victim that makes him/her to trust them..

So don't simply give you personal data.