Use free esxi and let the physical server running headless.
Run openmediavault 5 for NAS. You can expand its functionality by running docker container (transmission bt, plex, nms, web app whatever you want to try).
Run xpenology if you want to experience synology UI. Running xpenology in baremetal server require some degree of knowledge, run it from esxi is the easiest.
Why multiple VMs on a server?