If you perceive sex as just an activity u do..maybe you would not mind someone which lost his/her virginity.

But if u value sex as something u do on a man-women spiritual bonding thing..well then yes it matters.