It's so confusing and demoralising when you are unsure what to do for the best :(
The majority of thriving healthy babies start to gain weight after day four, and they are back to birthweight by now.
Their weight does not go 'up and down' - weight should be upwards.
They are not sleepy to the extent they need to be woken for feeds - but you are doing absolutely the right thing in waking him, and giving him ebm.
You can ask whichever hcp you trust the most about your next step - ask about breast compression (which keeps the baby feeding), switch feeding (which gets more volume of milk into the baby and also boosts supply) and feeding on both sides at least whenever possible.
The precise volumes of ebf are not important though - it's more important to keep feeding frequently, expressing when you can, and getting whatever ebm you get into him :)
It would also be good for someone who knows their stuff to really observe a feed and check milk transfer is effective.
I don't think they are right to be unconcerned - it does not sound like a case of 'panic stations' but a baby well below birthweight at two weeks with a history of slow weight gain who's sleepy is a baby whose feeding needs to get better.
Hope you get good real life help. Many babies in this situation turn around beautifully once these problems are addressed :)