It depends really on lots of factors, we have been very lucky and we bought lots of baby goody hampers with things like wipes, nappies, nappy bags etc...You can get great deals on these buying in bulk though.
I signed up to Boots parenting club and got a voucher for a changing bag. Its just simple and plain black, comes with a plastic changing mat but as I'm breast feeding It is perfect as I only need to chuck nappy stuff and a change of clothes and don't need to worry about bottles etc...
If you do end up breast feeding I would suggest investing in a decent pump down the line so budget for that if you wanted but don't spend lots on sterilizers etc as the easiest I've found are the microwave ones that come with the better pumps!
I'd agree with other posters about budgeting for easy to make treats and food for you. I bought lots of yoghurt and dried fruits and munched them lots as well as lots of nice juices etc. You need something easy to grab but something that is also going to keep your energy up! Cake is also pretty good. 
My little man has grown very quickly and is already in 3-6 month clothes so I would say go easy on any newborn clothes just in case and get lots of muslin cloths (I love them for everything) cleaning up dribble and sick, catching errant milk spray, a little extra layer, for privacy when feeding in public etc...you can't have too many in my eyes!
Try second hand free sites in your area or on FB, I got a load of clothes and second hand bottles for expressing etc that only needed a good sterilization and new teats.
Really though, newborns aren't too pricey. They are fairly happy with food, clean nappy and warmth!
Good luck and congratulations!