I usually begin this conversation with "let's have a think about what we'll do today", so that I'm not putting him on the spot there and then. If ideas come forth at once, then great! If not, we might do some brainstorming.
We also keep a list of things we might want to do, and places we might want to go, so we put them in where there's space.
We sometimes find it difficult to do things on the spur of the moment - we both like to wake up having some idea how the day will go, so we generally plan them a day or two in advance, there's less pressure that way.
If him not suggesting anything is bothering you, OP, then tell him it's bothering you. I have no patience for my DH seething about something I have or haven't done, if he fails communicate about it. If his body language is screaming "annoyance" at me, I've been known to say "if you don't tell me what's bothering you, I shall assume that nothing is wrong, so speak now, or forever hold your peace". Or in short, "speak up or shut up". Mind you, I do sometimes wonder if it's a symptom of being told to "stop moaning" as a child.