It's never 'U of Virginia' - that is 'U-V-A'.
UConn, UCLA, USC, NYU, and UIC, are the only universities I can think of off the top of my head, along with UVA, where the U is included in everyday references. I'll probably come up with more when I'm trying to fall asleep later...
The University of Pennsylvania is known as Penn. The University of Illinois is 'Illinois', and then 'Michigan', 'Iowa', 'Wisconsin', etc to refer to flagship state 'schools'. Similarly, 'Northwestern', 'Marquette', 'Loyola', 'William and Mary', etc with no U or University included in general references.
Then there are places like 'Mizzou' (University of Missouri)...
UConn's basketball team are the 'Huskies'. Geddit?