It just a lust on his part that could only be satisfy by making love with you. You on the other hand had some feeling for him. It is the thought of not wanting to be the third party in the relationship that restrained you. I you should stay away form him otherwise you will eventually give in to him.