I would be more concerned about the drugs and the gambling than the anxiety and depression - but all four in combination, unless well controlled to the extent that they are no longer an issue, would really make me hesitate, especially where children are involved.
There are posters who are vehement about MH issues, and will attack you for suggesting this might be an issue: however, as sufferer myself, I have made the decision not to date at all, because I know how difficult it is to support someone when they are ill, and how unreasonable and difficult I can be as a result of my depression. This came up last time I saw my psych, and he actually said that my decision was very sensible, and he whole-heartedly supported it (and he's not afraid to call me on bad decisions): I don't feel capable of making good decisions whilst I am ill, and I need to focus on getting well, even if it is a long and lonely process, before thinking about involving someone else in my life.
Of course, it depends on the extent of your friend's problems, too - a lot of people these days have a diagnosis of "depression and anxiety" and in reality, it's nothing more than the medicalisation of feelings well within the normal range. If so, that's not really a problem and easily handled by someone else. But if he suffers severely, and is not stable through treatment, then it will impact on both you and your children.
I think you're playing the right game at the moment - keep getting to know each other as friends, until you know whether or not you will be able to support him in his illnesses, and whether the drugs/gambling have gone for good: plus, if you're struggling to keep your hands off him, it'll be even better once you've really got to know him!