It sounds like hes really good at degrading your worth. I read your story before and it sounds like youre an awesome and strong person, and you should know and always remember that. You try your best as a mother, and Im sure youre doing the right thing for your family.
Stand up for yourself and your family. You should be the one laying down the rules, not him; make sure he knows that. Thats no way to be, for him to come and go; ignoring him completely is probably no way to go...eventually youll need to talk with him to work things out, but dont go with any suggestion that you dont want to.
As for right now, maybe its good to get some help or to take some time out to think about things. I know thats what i did with my sons dad -- he was good at degrading my worth -- one little phrase is vividly remembered in my mind, when I told him I deserved respect, he said 'no you dont. normal people deserve respect. prove yourself to me and ill respect you.' and i shut up. Well its no good shutting up and keeping silent, because i DO deserve respect and i AM a normal person -- and same goes for you, you and the children DESERVE RESPECT.
It took me a long time introspecting, blogging (this is where this site comes in), thinking about things, and even a few drunken nights out to just see me for a person whos worthy of love respect and everything else thats good in this world.
Until you can properly stand up to him, im sure youre doing the right thing by keeping him away. because it takes a lot of strength to stand up to something that put that much fear into you. I havent seen my ex for a very long time, and although I know my needs and the things i ask of him are not stupid, I still wonder what would happen if i had to talk to him again and stand up for my son and I.