You using closest area or single area AF? Try using single area and sticking to the center AF point, use focus lock and recompose if you want to shift the focus point to another part of the frame.