To put it bluntly - it's your fault for bumping his car. In all fairness, why should the victim settle for less when it was solely your mistake? In fact, even if you pay up to get his panels fixed he would be out of a usable car as it needs to be in the body shop to get it fixed to its initial state - and again, that's not fair for the victim at all.