I’ve had symptoms for 15 years now - bloating, reflux, variable bowel movements. Blood tests were normal but my GP agreed to refer me for a gastroscopy and later I had a coloscopy to investigate chronic abdo pain. Biopsies for Coeliac and colonoscopy were negative/normal. They diagnosed me with ‘functional dyspepsia’ and I have a hiatus hernia.
However I did push for a laparoscopy (chronic abdo pain with no obvious cause from scans was so awful it stopped me working) and I did get diagnosed with endometriosis. I definitely have worse symptoms around AF. So I have an IBS and endo diagnosis.
I know I’m sensitive to gluten - I did the low FODMAP diet where you exclude all foods likely to cause IBS under the guidance of a dietitian and then reintroduce them one at a time to see what is setting you off - but unfortunately I reacted to everything I introduced! Life was so bland without all of them. So I’m GF, try to keep my lactose intake to a minimum and it helps. I also get the aching joints, feeling rough, bloating, nausea and even vomiting at times from gluten intake but I’m not coeliac nor do I have IBD. As one consultant siad to me once ‘You’re all auto-immnuey, aren’t you?’ 
I also have asthma, eczema and migraines!
I think in your position I would push for a gastroscopy, you need it to be investigated. It may not be as worrying, as you suspect, but even the negative blood tests haven’t put your mind at ease. I think a biopsy and a good look down probably would.