Actually I read a book about Diana’s mother called “Frances, The Remarkable Story of Princess Diana’s Mother” in which the authors outlined how Johnnie Spencer’s job as an equerry required him to accompany the then relatively inexperienced QE2 on a long tour of the Commonwealth in 1954 which was a great success.
They go on to describe how the bond between the late Queen Elizabeth and Johnnie went back a long way and,
”… was a lot more relaxed than simply that between monarch and equerry. Prior to her marriage, Princess Elizabeth had occasionally written to Johnnie while he was on active service with his regiment during the war. There are, too, suggestions that he was at one time considered as a potential suitor. If that was true, then having once been close to being a royal consort may have ignited his ambition to marry as close to the monarchy as he could. It might also explain why the Queen kept her distance from Frances, considering her too much of a loose canon for her own taste, the differences in their personalities clouding the relationship.”
So I wouldn’t be dismissing what Charles Spencer says quite so readily, as he may be in possession of some of the late Queen’s letters to his father by the sound of it?