I think its a great idea.
For example, we are doing up my old house but hopefully won't need to put it on the market. Its a small terraced in a run down area and we were leaflet dropped by a housing association looking for homes to buy. Although the price will be lower than on the open market, it would save estate agents fees, and be an instant sale instead of me still having to pay council tax and insurance etc till its sold.
So that's one house that could be sold by leaflet drop.
When its sold we are almost certainly buying a house in a better area, to rent out. And that isn't going on the market either, its already owned by an work colleague of mine who is happy to wait till mine is done up and sold, because she is still getting rental income from it until we complete. (will be sold with tenant in).
A house on our street sold by word of mouth last year, someone happened to mention to a friend that her daughter was looking for a nice starter home and the grandmother of the person she was talking to was about to move into sheltered housing.
Moving with no Estate agent can save you a lot of fees, no chain etc. Although I would add some incentive in, like all legal fees to be paid by you etc, independent valuer to be used, as even with a swap you will still need to do it all properly.