I've done a decent amount of work on this both to satisfy my own curiosity and for work purposes.
I can't promise that they weren't listening in the past, but in the last 5 years or so, if you haven't granted an app microphone permissions, then it's not listening to you. It's coded in at the operating system level so almost impossible for app makers to work around, without resorting to particularly dodgy measures that'd be picked up by antivirus and there global community of nerdy privacy obsessed people very quickly.
Processing speech isn't a particularly simple task for a computer so it involves a fair bit of processing power, which would be visible either as local processing or power draw, or as internet usage for anything processed on the cloud. It's fairly easy to check whether the numbers go up when someone's talking or not.
A lot of the "Huh, I was talking about that and now it being advertised to me" is just very good statistical modelling coupled with big advertising networks and a healthy dose of confirmation bias. So Tesco's knows that 30% of white women in Wales with 2 kids (all of which they've worked out from your buying habits with a clubcard) buy this product 2 weeks after buying a different product. You're amazed at the fact that they've suddenly sent you a voucher for this product, but ignore the lack of coincidence for the other 4 vouchers for random crap you don't want.
Or Amazon know that a higher proportion of people than average paint their skirting boards in October. They also know via a tracking cookie that you bought wall paint from B&Q last week, so know you're a perfect candidate. The other million people who get shown skirting board paint don't even notice it, but you're now eyeing up your phone suspiciously.
And that's a simple example. Amazon might know you're a Married at First Sight fan because you went on the thread about it on Mumsnet and a tracking cookie caught it. It also knows you have Virgin Media, so knows what adverts you saw during it. You've forgotten about the advert for X perfume that was on that night, but it's sat in your subconcious and you tell your husband you'd like some for your birthday. Amazon also knows from your address details who your husband is, and also knows your birthday is coming up. Boom, let's send him some adverts for X perfume.
Computers find decoding human speech hard, but they find linking lots of tiny bits of data together to make a statistical model incredibly easy. So generally that's what happens.