I'm am in the same language position as your husband. As in I'm the second language user in the another country.
My son is now 5 and fully bilingual.. honestly there is nothing you can really do. The pressure is on your husband. He has to make polish a part of your son's life and enjoyment, otherwise it won't stick.
And to be brutally honest it will only get harder. At 3, he is having more time with you both, but by the time he goes to school.. all day at school = English, friends = English, tv = English, games and hobbies = English etc. it doesn't get any easier to keep another language going.
For me what I did was -
Make sure I have plenty of alone time with my son, especially things like extended walks from birth where I talked and talked his ear off in English.
Over half of our story books are English, and I will read to him English books every night
At least one long trip to England a year so he spends time in a native setting with his English family
Found tv shows and hobbies that he has only experienced in English. For example he has a few cartoons he watches, but he has only seen the English versions so can only use English to discuss the fantasy/name based parts of the show..
Tought him reading in English and now I'm working on writing, because research has shown that both cement second languages. The stat was that kids that can read and write in a language by 8 are significantly less likely to every forget that language when they are older, even if they aren't regularly using it.
Of course through all of that he must use English. But he has never complained about that because all of that is really fun! Make the second language an essential and fun part of his life, and you won't have to force it.
But if 90% of this life and the things he is interested in is in his first language, except the 10% of the time when he is forced to talk to dad in another language.. of course he will never be fluent.
In simple terms your husband needs to step into a very big and enjoyable role into his life and take his language as part of it.