Accommodation is fine - you can stay wherever you find is decent. Generally in Japan, all areas are fine. However Ginza is usually viewed as "atas"

You'll enjoy Ginza if you're there on a weekend. From 12 noon onwards they close off the Chuo Dori and only allow pedestrians to roam around the road.
On Day 2 - I can assure you that Yoyogi Park and Shinjuku Gyoen is different. One is a park, another one is a garden. If you go to the park on a weekend, it's very nice. You'll see a lot of people picnicking, strolling with their dogs, children playing badminton or running around, people working out in the area etc. You can bring food and find a decent spot to have your picnic there.
As for Shinjuku Gyoen, it's a Garden. You'll have to pay 200yen to enter the garden. It's huge with several different sections (Japanese garden, Chinese garden, French etc). For me, the garden was enjoyable because I went in autumn and the leaves were truly a magnificent sight.
So looking at your itinerary, I will actually suggest that you go to Yoyogi Park instead, as you'll be visiting Meiji shrine. Also, a park will be more enjoyable for a toddler rather than the Garden (just my opinion though).
Don't forget to try the Harajuku crepe in Takeshita Dori, and also drop by the Daiso there!
On Day 5 - As an alternative, I will recommend Odaiba.
"Odaiba, located on an artificial island in Tokyo Bay is an all-day destination that has several kid-friendly attractions including the Miraikan: National Museum of Emerging Science and Innovation, Daikanransha (Ferris Wheel), Sony Explora Science and Decks Tokyo Beach (restaurants)."
In Odaiba, there's also a trick art museum around, indoor amusement Joypolis, Fuji TV exhibits and the wonderful view of rainbow bridge + statue of Liberty on DECKS at night.
Excellent feedback.. sometimes reading the description online doesnt really prepare for what we will find there.