No it's not. I accepted a job offer over the phone and was sent a contract but then I went back on it as I got offered a job I thought was better (turned out I was wrong!). I don't think it's that uncommon tbh.
No, it's not too late. Simply call the company contact (e..g did your offer letter give the name of someone in HR you can contact?) and advise that, unfortunately, you 're no longer able to accept the role. (No need to give a reason if you don't want to, although they may ask.) Then follow up the call with an email to confirm what you said in the call, thank them for the opportunity and their time and wish them all the best for the future.