For someone travelling on budget, I can highly recommend World Scholar House in Taipei, website: http://www.worldscholarhouse.com/

It's located in a good area, very reasonably priced, easy to reach from the airport (read the instructions given on the website) and not too far from public transport.

Be careful if you want to choose another hostel as cheap accommodation tends to be really crappy in Taipei. Outside of Tapei 400-600 TWD will do for a basic but clean hotel room, easy to find if you speak Chinese (I don't). Backpacker accommodation rarely exists outside of Taipei and tends to be overprized.

I liked TaRoKo National Park a lot. Kaohsiung however is just a busy city with crazy traffic, nothing much to see. I also liked the flair in Keelung which is a short and cheap bus ride from Taipei (a bus departs close to World Scholar House) and DanShui (reachable by MRT) is also nice, both on the coast north of Taipei. I didn't go to Alishan but that's a place you shouldn't miss. No need to go to the big cities, they are not too different from any other city on the world. Including Tainan in my opinion, which is said to be nice though.

What I liked most in Taiwan and what makes it the nicest country I visited in East Asia so far are the lovely locals though. They are not really used to tourists, so they don't try to exploit or abuse you and rather have a sincere interest in meeting a foreigner and helping you in any way they can. I'm Westerner though, so my look differs to locals and maybe gave me extra-special treatment.

Cheers!