Quite right. To be an actuary the paper qualifications and experience components are very important, more than the "top school" factor that other fields (banking, finance, consulting, law) sometimes select for in addition to excellent results.

In fact, you'll notice many top schools don't even teach Actuarial Science! In Britain I can think of LSE and Warwick University, but they don't even have (for now) full exemptions! The major ones are City University (Cass Business School) and Kent. Although, if you did Mathematics at Cambridge University you might have as much of a shot for en entry level role as someone who did an Actuarial undergrad. So long as you show the willingness and potential to do exams. It's not about the degree - Accounting or Finance backgrounds are common too. In the US you can go to Wharton but it's not a straight Actuarial degree. Plus no major exemptions in the US anyway. NUS only has a joint Actuarial degree. Australia has quite a few.