Usually follow father side, i'm half chinese btw and since my father spoke cantonese to me ever since so i consider myself as cantonese. For your case, you may you're hokkien descendants and from Perak since your're born and raised there.