Ok. Been there done that, and I am begging you to STOP. You have to cut him off from you and let him go to court to get an order, this way you will have an oppurtunity to talk to the Judge yourself. Most men are afraid of that because that is when their skeletons come out of the closet. However, if you keep sleeping with him then it looks like you are accepting what he is doing. As women we must remember that men are only going to treat us and our children the way WE let them.
Decide what you want then go from there, if this is something that you do not want then you need to make some new arrangements. It is your house and if you do not want him to come over after a certain time then YOU have to STOP that. You do not have to cater to him, but for the sake of your son you are going to have come to some type of co-operative arrangement.
My daughter too has medical problems and her father would choose to come and see her once every couple years. When she was in a coma for 36 hours due to her health he never showed up, and 6 weeks after she came home from the hospital he called and told me it was my fauly because i did not allow him to see her when he had time.
Men like him and presumably your son's father only want to be fathers when it is convenient for them. I don't like drama and I do not like being manipulated and lies too. I got tired and stopped his 1 every two year visits that is why he is upset with me, that and i refuse to sleep with a man who likes to sleep around. I am a firm believer in if i allow you to treat me like nothing then you will.
Live your life and take care of your son, and do not let a father who wants his cake and obviously eat it too have what he wants when he wants. The child's needs must always come first, but is his father a good example of the kind of man that you want your son to be. That is something to think about.