It's never a bad thing to pay back.

You could also highlight to them about the allowance and because they owe you 'x' amount of allowance, you felt that there was no need to refund or pay back due to outstanding allowance that was promised.

Just ensure that your contract did state the allowance and amounting to how much per months (without special terms and conditions applied). If they insist you refund and yet not provide you your allowance, you can bring this to court in which I'm not sure if its worth your time and money because you will be looking for a new job if you went this far.