Exactly. It's poh-ka, or poe-ka dots, not pole-ka or poll-ka.
Just as the nuts are ah-monds, not AL-monds. Oh heck, almonds are probably not real nuts at all, and this is Pedants' Corner... It's okay, I'm prepared for a roasting.
I'd only wear polka-dots under extreme duress, but if I absolutely had to (hostage situation involving forced pantomime, for example) I'd avoid the pronunciation issue by saying dotty or spotty.
One of my (many) pronunciation rages is the hard "G" in the middle of ringing, singing, bringing: ring-Ging, sing-Ging, brin-Ging all drive me nuts. Yes it's probably regional, but it's teeth-grindingly annoying to me.
There, instead of their or they're. Dose instead of does. We was, instead of we were. Should of instead of should have. All the usual suspects...