I spent a week in San Diego last month with my partner and 1 year old. For public transport reasons, we stayed Downtown so that's a good bet.
Please be conscious that there are a lot of homeless people and the difference between staying on one block Vs another one along can sometimes be huge. We based ourselves on Broadway at the US Grant Hotel which was fantastic and we upgraded at checkin for 50USD per night and ended up in an enormous suite. Book a basic room direct through Marriott's website, sign up for Convoy, and closer to your stay they have a guy who will email and offer you discount upgrade options too.
From that hotel, and any on Broadway you can get the bus right in front of the hotel to go to Balboa Park which is where San Diego Zoo is. You can also walk safely to Little Italy for dinner. You will make several trips to the park and to Little Italy for sure! Downtown places for breakfast and lunch: Swami's cafe, Corner Bakery Cafe. Balboa Park lunch (and cocktails!) in Panama 66.
Getting to Sea World from Downtown is also straightforward enough, especially if you have a car (but we did it on public transport in less than an hour). Get the all day dining tickets online ahead of time because even though the food is horrific, it's mega expensive to pay as you go! Last orca show is at 6pm and is less busy than the earlier afternoon one. Sit in the shade and take hats.
The New Children's Museum others have mentioned is a must-do also. I've never been in a museum quite like it. Every single child in there looks so happy and inquisitive.
The Fleet Science Centre (Balboa Park) is great too.
Spend a day on Coronado Island. Hire bikes and hit the beach. Again, bus ride, or a bus and a lovely ferry, from the US Midway hotel is very easy.
Always buy tickets for attractions ahead of time and download the Pronto app for public transport purposes. Makes it super easy.
Avoid Gaslamp quarter, especially with children, for meals and accommodation. I saw, for example, a man smoking a crack pipe, leaning against a shop window in broad daylight on the main street on a weekday afternoon.