My DH and I went through IVF before we applied to adopt and then eventually adopted our 2 DDs. It was more clear cut for us, as there were no embryos, after only one immature egg was harvested. I was told that there was only a 1-5% chance of IVF leading to a healthy baby, so that settled it.
We had a preliminary interview with 2 social workers, who told us that we had to take time to come to terms with not having our own bio DC. We had to agree to go through therapy, which helped and the following year we were ready. I knew I was ready, as my DSis told me she was pregnant and I was just delighted for her, not jealous at all.
Adoption social workers have to be sure that you're not going to change your mind sometime down the line and, as you have those frozen embryos, they're bound to have concerns. (Although, from my experience with social workers, they're not exactly the most tactful of people so I can understand you being upset.
You need to have a thick skin if you want to get through the process, but if it's right for you it really is worth it, I wouldn't be without our 2 DDs now.