in this case, the book relates to lots of visitors, hence visitors' being more correct.
If the book pertained only to one visitor, ie it was the personal possession of one single visitor, one they had left at your house maybe, then "Visitor's book"
Would you like to write in our visitors' book?
Would you like to write in our visitors book?
Both ok.
But:
Could you remember to return the visitor's book when you see him?
Only this is correct.