Morning, I also have a large 5-weeker, and yes they are big heavy lumps! Although they are obviously not going to get lighter, my experience with DS1 was that as their muscle tone improves and they hang onto you a bit more, they become easier to move around. Also, your own muscles really do get stronger. In any case, babies are much more robust than you think, my toddler is a little over-enthusiastic with hugs etc, baby really doesn't seem to mind too much :)
Re the breastfeeding, try to chill out and be confident in your own body. Don't know what you mean by going downhill, but if you mean your breasts don't feel so big and full, then that is entirely normal and a sign that your body is adjusting to perfectly meet the needs of your baby. You will be producing enough, until the baby decides he needs a faster supply, and then you will get a day or 2 of intensive feeding which your body will respond to.
If you mean your DS is feeding a lot and isn't satisfied, you maybe are going through one of these spurts. Relax, go to bed with your baby, feed on demand with lots of skin to skin an and a nice hot chocolate! If you don't have other pressing things to do, these days are lovely!
New babies are indeed stressful. I do think 6 weeks is often a turning point though. Hang in there, try to stay relaxed and as long as your DS is doing plenty of nappies and growing well, then sounds like you are doing great.
Re man-flu: make sympathetic noises, but he'll have to look after himself. You really want to avoid catching it. Being up all night feeding whilst being ill is really no fun at all!