The point is are you confident to handle the situation if the situation deteriorate.
eg.
1.He delay some more, dunno when maybe 2 years then he lose interest. They your 200/month penalty will be irrelevant.
2.Can you get another contractor to continue and complete the work with the same balance cost?
3. Both of you get mean and start @#$%^&*() with each other, can handle the consequences?

These horror stories are splattered all over this forum.

Good Luck..