Probably something to do with some sort of resentment he has with you neglecting him in favor of the kids, which is fairly common; mind, this is HIS perception, not necessarily the truth - but it is valid to HIM, and therefore you have to address it.