Does this happen when you use a certain player?
How are you turning it down. Most players have a volume control, but Windows also has one. If you can't see it in the system tray (bottom right) then go to control panel and open 'Sounds and Audio Devices' and tick the box for 'Show volume control in taskbar'