I really think your dh needs to reasses his attitude here drholnes. He can't adopt without being accessed.
Would he want others to be able to patent vulnerable children just on their say so.
The process is almost as much for you as for the child (IMHO) because it enables you to think about vital things like:
How you were parented
How you are with children, your own , other people's children etc
Your relationship and how you will handle having a new child or children in your home and lives
Towards the end of the process your home is accessed to see if it will be a safe place for children. Despite having a birth child who was safe and well, we (like all prospective adopters) had to be accessed for this.
We also learnt about our own support network, which is useful.
If this is the only reason your dh has issues with your social worker, I think he needs to access his attitude and get to work.
There are currently more people wishing to adopt than children to be adopted, so I am led to believe.
The one couple I know who faught the process a lot ended up going through adoption placement and it fell through.
The process is there for a reason, you will not beat it! But if you relax, and by you I mean your dh and take it on board you may find it is useful.
Good luck 