I'm surprised you've never seen one at an aquarium or zoo. But it's not that crazy that you didn't know. You can pet them at our local aquarium. It's not that exciting. The stingrays are more fun to pet.
This is so embarrassing to admit, but I think I have actually seen one before on holiday (ages ago, when I was about six or seven). I was snorkelling, and saw (what I thought was) a sea cucumber. It looked to me like it was rooted into the sea bed and just flailing around along with the current and waves, so I just thought that was clear proof that it was a vegetable