I remember Alan Titchmarsh suggesting a monthly visit to a garden centre to buy a plant in flower, as a way of ensuring that every month has something in flower...
How important is fragrance? Some shrubs have beautiful flowers that bees/pollinators love, such as Ceanothus, but the flowers don't smell great (to my nose anyway). It's still worth having because its appearance and usefulness mitigate the lack of nice scent. I have a Philadelphus "Innocence" which is variegated, so less vigorous than regular green varieties (important if space is an factor) but the orange blossom scent mentioned by a PP is fabulous.
Some phloxes are marvellously scented, others less so. You could look for a variety you like with good scent here. Fragrance is really important to me too and a lot of my garden plants have been selected for that reason, although some are just because they look attractive or attract bees and butterflies 
Filling in gaps with Oriental trumpet lilies would be great, and as already suggested, pots of annuals such as stocks, night scented stock, heliotrope, petunias and nemesia will top up scent into autumn, brilliant when the weather's nice enough to sit out!
(Your Jasmine might thrive better in the ground rather than a pot. Being in a pot restricts root growth, which can have an impact on the top growth. If it's possible to stick them in the ground, I would. Some plants do manage very well in pots, but ones that are inclined to get reasonably big will tend to struggle more and perhaps be more susceptible to diseases/pests etc)