I had similar issues with my newborn, I even had a health nurse come to our house and visit us. She didn't even wake our baby (or try to) even though I was worried that she was not waking properly.
She looked at her and said she was nice and pink so would be fine.
About an hour after she left the hospital called back to say the lactation consultant had an appointment in 20 minutes time so if we could make the appointment they would see us.
We raced in to the hospital and the lactation consultant took one look at our daughter and walked us over to emergency, insisted we be seen straight away.
Our daughter was severely dehydrated, they said her heart rate should have been round 140bpm but had slowed to 100bpm.
If we hadn't had that call from the hospital our daughter likely could have died or become extremely unwell overnight despite the midwife who visited telling me she was fine.
I went against my instincts and believed that MW, never again will I do that. If you feel something isn't right, follow it up, if you feel that opinion wasn't right then seek another one. Don't feel bad about it, it may be the thing that gets your child the treatment they need.