As far as I know it's possible. I have a Vita and a PSTV, I just pop it out and switch I think, the first time I try, it works.

If it doesn't work you can backup your games and saves on the PC. Content Manager Assistant.