Maybe you should ask yourself, do you want to stand out from the other interviewee? Just because other people doesn't wear it, doesn't means it's the right way.

Most interviewer expect you to dress formally for all the interviews, even though they themselves might not have dressed formally. This would show them that you are really serious about the job and it would give good impression on the interviewer. It better to overdressed than underdressed.

Furthermore, stop giving excuses. It would irritated the interviewer even more and shows that you are not a manageable employee. Just be honest, sincere and apologize for not buttoning your sleeve. If you don't know the answer to some question, just say you don't know and said you would try to find out about it if you get the chance.

One last thing is try to talk in proper english and write in proper english (if the interview is in English). You would want to show them you are businesslike and able to speak clearly and properly.

Good luck in future interviews.