trying to find well-written challenging fiction that isn't widely known

The flowers of Hiroshima by Edita Morris

The Royal Game by Stefan Zweig

Both not well known, but definitely literary treasures!
I think the Edita Morris book is out of print, but you should be able to get a good copy for around £10.

Geek Love by Katherine Dunn?

Late to this party but what an amazing set of titles! I’d add We That Are Young by Preti Taneja, a version of Lear set in contemporary Delhi. And anything by Muriel Spark, so ruthlessly and enjoyably unsentimental. And anything on Backlisted, as someone mentioned above, is worth a read. Jane Gardam and Barbara Comyns were my finds of last year.

Their eyes were watching god. Zora Neale Hurston

Steinbeck: east of Eden
Dos passos : USA
Mailer: executioner s song
Atwood: the blind assassin
Balzac: la comedie humaine/ human comedy
Hannah Kent: burial rites
Flaubert: Madame Bovary
Capote: in cold blood

Thomas Pynchon - all of them are brilliant and challenging, The Crying of Lot 49 is a good place to start
Don Delillo - Libra
Neal Stephenson - Baroque Cycle
Umberto Eco - Foucault's Pendulum
George Perec - Life A User's Manual
Joseph Heller - Something Happened

Oh, and not fiction, but a volume called 'Oscar's Books' about the fate of Oscar Wilde's library (sold off through a window in disposal of his house and belongings) is fascinating. The author is very well-read, and has attempted many, but says that he isn't capable of reading even the ones that are known to have belonged to that collection (i.e. many classics and translations).

Potentially some of the 'New Stories from the Mabinogion' series - Welsh authors doing a contemporary take on some of the ancient ones and coordinated by Cardiff's School of Welsh. Possibly a good modern 'translation' of the Mabonogion too. Some of that series are great - White Ravens is one of my favourite novellas ever.

Would any of these suit?

Fugitive Pieces - Anne Michaels
A Hero of Our Time - Lermontov
Le Testament Francais - Andrei Makine
The Scarlet Letter - Nathaniel Hawthorne
Night and Day - Virginia Wolf
Ann Veronica - H.G Wells
The Sound and the Fury - William Faulkner
The Heart of the Matter - Graham Greene
By Grand Central Station I Lay Down and Wept - Elizabeth Smart
A Journal of the Plague Year - Daniel Defoe
Reflections in a Golden Eye - Carson McCullers
Rhine Journey - Ann Schlee

All great books

There's some amazing SF out there at the moment, & if it's not a genre she usually reads it might tick your boxes for something she's unlikely to have read. The Three Body Problem by Cixin Liu recommended above is an excellent read (though if she does read SF it's very widely known & popular).

But given your criteria of "a theme or a narrative that she has to wrestle with" I'd suggest Too Like the Lightning by Ada Palmer. It's set in a 25th century where ultra-rapid transport has made geography pretty much irrelevant, and people belong to 'Hives' based on differing value structures, each with their own city-states and laws that adherents and visitors must follow (there are also 'white-laws' who follow all laws, and 'black-laws' who are essentially anarchists whose territories have only a pared down socially agreed minimum of law. It's full of references to the Enlightenment, discussions of philosophy, religion & history & how people get & keep power. Definitely not an easy read, but very rewarding.

Was coming on to suggest looking at Persephone’s booklist but I’ve been beaten to it.
The Mint Lawn by Gillian Mears, Australian author.
Alice Munro short stories
Elena Ferrante
Angela Carter

Agree with the suggestion of Laxness above - Independent People is a wonderful book. Probably too well known, but the book that's made the biggest impression on me recently is Haweswater. Another one I often recommend which is (I think) a beautifully written little book but not well known is The Short Day Dying by Peter Hobbs, about a lay preacher in Victorian Cornwall. He also wrote another called In the Orchard The Swallows, which is set in Pakistan, so may be of interest if not already read?

Carol Birch's novels are very different, and have some unusual themes. Try Jamrach's Menagerie, or Orphans Of The Carnival.

In translation, I’ve recently loved Magda Szabo’s The Door and Abigail (Hungarian). How well-read is she likely to be in Irish fiction? I would recommend anything by Anne Enright, Kevin Barry, John McGahern, Edna O’Brien, Colm Toibin, William Trevor, Elizabeth Bowen (short stories as well as novels in all cases) — more immediately contemporary recommendations: Mike McCormack’s Solar Bones, Claire-Louise Bennett’s Pond, Eimear McBride’s A Girl is a Half-Formed Thing or The Lesser Bohemians, Sara Baume’sSpill Simmer Falter Wither, Belinda McKeown’s Solace, Mary Costello’s The China Factory, Elaine Feeney’s As You Were, Rob Boyle’s Threshold.

I don't have an answer for you, but I'm intrigued that you specified "challenging."

So am I. What do you mean by challenging? Shakespeare? Chaucer? Tolstoy, specifically War and Peace?

Willa Cather is an author who is often unfairly overlooked I feel. My Antonia is a wonderful book.

John Cowper Powys. Challenging. Absolutely brilliant. Almost forgotten.

I don't have an answer for you, but I'm intrigued that you specified "challenging."

Dorothy Dunnett Lymond chronicles are an excellent read, start with Game of Kings

Alan Paton's Cry the Beloved Country is of course seminal, but she might not have come across it.

In that case, more names I'd pick out from the AWS are:
Nadine Gordimer
Yvonne Vera
Chenjerai Hove

Of the last two I've only read their poetry, but their novels are highly respected.

I treated myself to membership of the Unbound book club recently. You get one of their titles each month as an ebook plus a few other perks. Their range is very varied and you're supporting developing writers.