Hi OP
You sound lovely.
I’d pop round with some flowers (maybe from your garden), a cake and a card with your names (including your baby) and number on, introducing yourself. Maybe take your baby with you when you go. Put in the card that you are at home a lot and would love it if she popped in for a cup of tea sometime.
She may be British and/or speak perfect English. But, if she doesn’t, it’s still possible to ‘chat’ to someone by both of you speaking in your own language and lots of hand gestures. I’ve done this many times. I once had a two-hour chat about all sorts of things with a lady I was staying with in Moscow, when neither of us spoke more than ‘yes’ and ‘no’ of each other’s language. And I regularly used to chat to my son’s friend’s mum at the school gate, who spoke Somali and Italian but very little English. When I lived in another country, my neighbours always invited me in to chat. It was embarrassing for me (and them) at first, as I couldn’t converse much, but it helped me learn in the long-run. And I was so grateful to them for keep inviting me. We’re still in touch nearly 20 years later.
You have something in common if you both have babies, so may be able to swap baby clothes or help each other out in other ways. Then, when the kids are a little older, they could play together.
As for how people who are Muslim behave, there’s some nonsense on this thread. There’s no such thing! How do people of Christian faith and/or culture behave? The answer is ‘depends’ - on where you’re from in the world, your social class and (more to the point) what your personality’s like.
Nationality is more a factor than religion, so if she has a foreign accent, ask where she’s from, both the country and the region. If they’re recent immigrants, especially refugees, she may be very shy and still recovering from a lot of trauma. But having worked with refugees for decades, I’m sure she’d be very grateful for you being friendly and hospitable, even it’s a bit excruciating for you both because she can’t speak English or if it’s not a great time for her (depression etc). People can feel so unwelcome and isolated that the tiniest of gestures makes a huge difference.
If you get no answer or response from the card and gifts, don’t give up. Some people are very shy and feel awkward. It may be worth your DP having a chat with her fella when he sees him and say you’d like to be friends.
Good luck