The problem is because you're feathering the edges. When feathering, it does what it says, blurring the edges of the selection.

For rounded edges, select the area you want, then choose: Select -> Modify -> Smooth

The sample radius defines the size of the curved edge.