It's worth knowing that this is an issue that mainly affects girls. Lack of confidence is a huge issue, and research also shows that girls have a tendency to underestimate their abilities in relation to their peers (imposter syndrome!). You say DD is doing well in tests so is in the right set for her ability. That she struggled with that homework and struggles with some of the tougher work is fine. Maths isn't supposed to be easy all the time, especially in top set if the teacher is doing their job properly.
Spending 3 hours on homework isn't necessarily the best course of action if she's stuck. Spending an hour on it, then asking the teacher for a bit more help could have saved the time. Girls with a tendency towards perfectionism find failing to do something or leaving something unfinished quite difficult to do.
You say that it doesn't help that she's sat next to a condescending maths whizz. A first solution rather than moving sets would be to move her next to a more supportive partner. This could be someone of a similar ability (I have two weaker girls in my top set who like working through things and asking for help together), or someone good at maths but helpful (another weaker girl sits next to one of the really bright ones who explains stuff to her).
Definitely approach the teacher, but not to request a set change in the first instance, but to discuss how your DD is feeling and asking about seating changes. It could be that your DD is really struggling and then a set change could be broached, but it could be just that she needs a bit more support, reassurance and better tactics for what to do when she doesn't understand something.
Like I said, it's worth knowing that this could be part of a nationally recognised issue with girls in STEM subjects, which contributes eventually to the gender pay gap.