Hi Chloe!
I used to be an infant swimming teacher. I would say that 5 weeks is still a bit young - I think that 8-10 weeks is more realistic. You also need to take size into account, and, really, baby should be at least 10lb.
The pool needs to be nice and warm or ds will get cold quite quickly. You could put a tightish vest on over his swim nappy to help keep him a bit warmer (please make sure he wears a swim nappy!). For a first visit, 15 minutes is probably enough. Try to choose a good time when he isn't tired or hungry.
Make sure you smile at him and re-assure him a lot, especially if he doesn't seem to like it. (In my experience, 80 - 90% of very young babies who don't like 'swimming' have a very anxious parent with them, and I tend to think they pick up their anxiety from their parent).
Good idea to take him in the bath with you, and let him get used to having water over his face. Try to dribble it over his forehead from above, rather than splashing it upwards which can lead to water getting up his nose. You could also let him swish around in the water, holding him under the arms, or float on his back (obviously, you will need to support him under his head for this one).
BTW, I remember your thread from breast and bottle feeding a while back. Have things started to settle down a bit now?