There are a couple of settings under the Wireless-Professional tab that you can disable to improve performance. Example Airtime Fairness, Roaming Assistant etc.

The other issue is the RT-AC3200 (not the GT-AC5300) has the older Broadcom 5GHz chip and a slower processor than the GT-AC5300 or RT-AC86U.

You may consider adding another one or more AP from Ubiquiti or Mikrotik and let these AP handle the 100 connected clients, which obviously the AC5300 cannot handle. ASUS had not published the max client for AC5300 and guess its probably 30 to 50, or so clients.

If the ASUS works fine, then adding one or two AP will be sufficient. The enterprise AP are not that expensive and will support a lot of users each. With 100 concurrent clients, probably get 2 AP, to load balance and also to offer redundancy. Prices are couple hundred each.

You can still keep AiProtection, VPN client and VPN server etc from ASUS.

You are probably right about Mikrotik however ubiquiti is ranked equal. IF you want to get serious enterprise stuff, thousands of clients, VOIP, hardened security, realtime transactions routing, fault tolerant etc neither are in that ballpark, not even close.

At times, the discussion that "SME" gear is best and high end consumer gear is crap mindset is misplaced (not referring to you here, just saying in general). The main difference is that if you do not require multiple VLANS etc its simpler to pick the consumer gear as once setup, you do not need to spend time monitoring and tweaking.

The RM500 router, if the right model with good reliability including firmware is sufficient for whatever Unifi package at the moment, or a bit of future proofing the RM1,500 or more router with more fancy features would be enough for up to 1gbps fibre. Just look south at the SG networking forum and see what is common there.