tinkly - yes, it's a really big worry, as I understand it.
English Lit is inextricable from cultural context. You can't simply pretend that centuries of literature being written in English, by a particular group of people, didn't happen. Nor even that literature written in English still tends to be dominated by certain voices. You can try, by choosing more authors who are not straight, white, middle- or upper-class men writing in standard English, but it's tricky.
And I do think interpreting poetry can't be 'nailed down'. It'd be bad interpretation if it were.
The question about what the author is 'getting at' is partly political: basically, back in the 1970s, people started theorising that probably the author's intentions aren't the most important thing in the world - because, obviously, an author might write a line and only subconsciously be aware of some of the connotations it has. Plus, students can't be expected to know everything about what the author thought or meant.
So, it matters more that the interpretation is tenable.