In what conditions the deposit is not refundable?
1) if they are not appointed as the main contractor
2) if they have not receive the renovation deposit.
My interpretation solely rely on the phrase.

Did your main contractor has b&w written quotation stated with t&c " The refund of the design fee will only be eligible if we were appointed as main contractor for all scopes " as he said to you.

if no b&w, it will be hard for you to argue with him.

That's the definition of " all construction scopes " makes the argument then.

Most design and build firms will separate design fee and the construction quotation, they will ask for upfront fee for design drawings

Whether they will return these design fee for client is depend on their company policy, some of them will return  or deduct the design fee from the total construction cost when you decide to appoint them for construction as per their quotation.

Some of company will not deduct the design fee even if you appoint them for construction works later part.

Yet your main contractor ask for deposit, is this deposit is part of the construction cost or as a design fee?

This you will need to clarify with him upfront before they actually start the design process.

I would say the scope of work of main contractor shall confine to what he has designed during the design stage.