Yes, the whole point is to get out of your comfort zone and to get used to public speaking.
What I meant in my previuos post is that if you don't like how things are done (or timing or location) in one Toastmasters club you can always attend as a guest at another location.

Give it a try, most of the people I've met from Toastmasters are very sociable and helpful. They're also very supportive of newbies.