I wouldn't poke a pin right there unless it's an eyelet collar. Some say that it wouldn't leave a mark but I guess you'll have to poke at the same point in order not to damage the interlacing.
Some collar pins look just like normal safety pins.

Some collar bar I thrifted from ebay. I find that knit ties and collar bars work well together as the knitted tie steps down the formality of the collar bar. Forget about the red sweater. I hate it as well. Used to wear it with my gray half-zipped stand-up collar sweater but couldn't find it in my closet. Anyway I think a collar bar/pin hardly works without a jacket/vest or sweater.