Well, I think you are being slightly unreasonable.
I understand what you are saying re wanting to be where you grew up it, it would be great to be able to live where you want to- it's frustrating to not be able to afford what you'd like (especially when you are asking for a modest home), but having what you want, where you want it really isn't a right.
I also agree that rents in the SE are very high (think it's madness, personally) and this makes it hard for those on low incomes. However, it is always about supply vs demand- and what the market will bear. There must be sufficient people who want to live in Brighton and can afford to pay those rents, or the rents would be lowered. The state could do more to provide affordable homes, but essentially this is about cutting your cloth according to your means. Your means are limited, so you need to find a way to make the best of it.
You do have choices- none will give you exactly what you want (or at least not immediately), but nonetheless they are choices.
You could:
a) move further out from your ideal area (you said 10 miles or so) to afford what you want. The compromise here is the commute/cost of commute/not being where you want to be and further from family.
b) stay in your preferred area but accept you'll have to be in a smaller/less suitable property. the compromise here is the size of property but you don't have commuting issues/costs and are where you like to be and are close to your family.
c) one or both of you take on some form of education or training so that you can improve your earning capacity and in time can perhaps afford what you want, where you want it. The compromise is having to stay put (or choose option a whilst doing this and looking to move back later) and waiting, also having to put in extra work which will be hard whilst working/having a young child- but is probably doable. E.g. distance learning/OU or night classes)
d)investigate moving somewhere further away with lower house prices/rental costs- if you can both be sure of getting work. You may be able to achieve a better standard of living but the compromise is being further away from family/friends/Brighton- you may find your prefer where you end up, or you may not.
Ultimately, you and DP have to find the compromise which works best for you.