Some independent advice is rarely a bad idea. What you are suggesting is not necessarily a bad idea but for me I'd say it really depends on a number of very personal factors to you. You don't have to answer the questions below but these are the kinds of consideration I mean?
-Where do you live now, owned or rented accommodation and if owned, mortgaged or owned outright and how much equity will you have when you retire, is your current home in an area that should have average or better capital growth. What is your mortgage interest rate (if you have one) and if fixed, how long until renewal, any early repayment charge? If you have a large expensive mortgage you might be better off using the money to overpay or pay off the mortgage on current house rather than buying somewhere additional.
-What will your £300k buy you in 'retirement' area, will you need a mortgage to get the kind of property you want and what will the rate on this be, how much do you expect house prices in this area to increase between now and retirement? How easily will the kind of property you buy rent out and for what rental yield once costs (including tax, agents fees, maintenance) are deducted? Bear in mind the kind of property that's high in demand for rental might not be the same as what you would eventually want to live in.
-What's your income and tax band (if you are both 40% tax payers having a rental property can be expensive as you pay so much in tax on the income), how long until retirement, how is the rest of your financial situation? So many people buy a BTL as their 'pension fund' but seemingly ignore the possibilities (which can be much more tax efficient and less risky) of putting the money into an actual pension scheme. Have you already maxed out all ISA allowances for you both?
-What's your risk appetite with regard to this money, property is usually a fairly safe investment but not totally risk free and if you own your own home already it's likely a fair amount of your capital is already tied up in property market. If you really couldn't bear it if house prices crashed and you lost some of the inheritance how would you feel? Equally if your risk appetite is a little higher you could make a better return by investing in well managed funds (I wouldn't advise just playing the stock market unless you know what you are doing and certainly not crypto or any other dodgy 'investments' but you can pay a small fee to have someone else manage a balanced portfolio for you).
-Do you have any DC and are you likely at any point before retirement to want to access your capital to help them out with uni costs, house deposit etc? Or any other major capital needs on the horizon, if so you probably don't want all your money tied up in property as this makes it very illiquid
-Finally there is the emotional and practical side of being a landlord, personally I would always advise against letting out any property you have an emotional attachment to, it's not always the case of course but you have to assume even carefully chosen responsible tenants will not treat any property as nicely as you would (e.g. if you buy somewhere with a lovely mature garden it's unlikely they're going to spend hours lovingly tending it when it's not theirs at the end of the day, they may not even want to allow a gardener paid for by you in to maintain if doing so is disruptive for them) and in the worst case you can end up with bad tenants who absolutely trash the place. If you've bought a beautiful 'dream' retirement house with e.g. original period features or high spec modern kitchens and bathrooms, how would you feel about them being damaged or even destroyed? You'll likely as a bare minimum have to do some cosmetic works to a property that's been rented out for a significant period of time before you move in, is this OK by you or too much hassle? Equally what about maintenance to keep the property safe and suitable for tenants, obviously you can pay for all this to be done by agents (you will probably have to pay for at least some kind of management service if you don't live locally) but this does really eat into your profit margin.
Basically my advice always is with buy to let, do some sums, think carefully and dispassionately about it, consider both the best and worst case scenarios, take advice and don't assume it's either BTL or nothing, there are lots of different ways of investing your money that should have the same outcome of you being able to afford to retire when and where you want but different routes to get there, IFSWIM?