A really good spot around 10 mins from the M4 is Active Tots http://www.activetots.org/ My 5 month old loves it but it's suitable for kids up to the age of 4! Dinton Pastures is also nice and buggy / kid friendly. There's a nice cafe too. http://www.wokingham.gov.uk/parks/parks/country-parks/dinton/
I would recommend the River and Rowing museum in Henley-on-Thames. North off the M4 at junction 8/9, easy to get to from Oxford.
Lovely museum, great permanent Wind in the Willows exhibition, good cafe and facilities, good parking. You can walk along the river into the town centre and there's a playground 2 minutes from the museum on the river front, large grass area between museum and river to picnic on too. www.rrm.co.uk/