There are many things that are my unquestionable legal right to do. But just being allowed to do a thing is very rarely by itself a solid moral justification for doing it.
The moral justification is that by giving permission, he would create an obligation to the child he might not be able to fulfill. What the woman wants is secundary, he has to decide on the basis of his responsibility towards the child.