Seriously, get him to go for blood checkup. You are entitled to a safe mind that he is healthy and fit.

secondly, can you trust him ? If you think you can trust him, why not ?
If there is no trust, there is no ground for the relationship to work anymore.
Everytime you see him with another girl, you would be very suspicious.

oh.. he never really tells you ?
this is problematic. If you confront him about it, he would take that as an infringe of his privacy ( diary)

could it be just his fantasy ?

Try talk to him about it.

ye know, one word of advice for you. You should know what kind of a person he is by now.
Not that i am throwing cold water at you, but you must be prepared for some possibilities in the future.
some people never change. It's true some businessman continues to visit hookers even when they had their own wife and children.

IF you love him, you will just have to loves his flaws as well. And as long as he loves you the most.
If not, better opt out soon before it's too late to calls it off.