I think in this case maybe the left midfielder or one of the defensive midfield need to track back because they let you solely doing the defending. Or maybe you can improve your anticipation skill. I'm not an expert but a good defender can read and intercept pass and opponent movement. What are your role in fullback? Are you an attacking like Evra or more of defending like fullback in Holland national team?