Was that just a one-time occurrence? If so, then I’d brush it off and c’est la vie.

If it does happen often, then it could be he’s not that considerate of your feelings. You can have a heart-to-heart chat with him then.