Most of the time, employer have their self-interest only if they want you to sign non-compete agreement. They don't understand situation might change, employee might jump to another competitor company to prolong his expertise and profession. Exchange of skill and learning from each other is a norm nowadays.

On the other way, employer can protect themselves, if their design is copied 100% directly on another company, causing lost of their existing business, and lost of reputation if those design has gone wrong. By that way, case could be investigated, and compensation or forced resignation by law, is deemed reasonable.

However, no worry. Continue to work as your profession in the competitor company, no need to tell anyone too detailed about past job experience. Don't tell your former colleague where you are now. As long as you work professionally, never misuse info and design from previous companies blindly, then it should be ok.