As I'm sure you already know, it is fast and you shift your balance around an awful lot. If you're fairly pregnant and your cente of gravity has shifted, I think I'd give it a miss as you'll get involved with the game and find yourself shifting balance quickly without thinking about it and might find yourself unbalanced by your bump. If you're not far enough along to have your centre of gravity altered much then I'd think about going if I were playing on a pony I knew well and was reasonably trustworthy.
One of my friends is seriously into her polo and she played up until her bump altered her balance but other friends stopped riding the second they found out they were pregnant. I rode until 6 1/2 months but flatwork/hacking only. I think it's up to you to decide what you feel physically and mentally comfortable with.