First off, the overwhelming majority of men who father children do a reasonable job of looking after them, either providing for them, or caring for them. Almost everyone you know is alive because his or her parents cared for them, and most children are still raised by couples.
Very few parents, are systematically cruel to their children (unless of course you make the extreme value judgement that being brought up in the spectrum of acceptable ways is an act of cruelty, which strikes me as a posture).
Further, not only are very few adults cruel to children, but very few indeed ever have sexual relations with children under the age of puberty, which is very exceptional. This is both a social taboo, but also an internalised psychological attitude, and quite likely (suggests Levi Struss on incest taboo) a condition of human culture.
The reason that the incest and paedophile cases are quite as shocking as they are, is because people are overwhelmingly repulsed by them.
There is a complication that taboos also draw a degree of eroticisation among people with a great deal of leisure time (an aristocratic privilege which has trickled down). But even taking that into account, eroticisation of pre-pubescent children is widely condemned, and it seems likely genuinely repulsive to most.
Lurid fears about paedophilia, taking advantage of the very strength of this taboo, have led to hundreds, if not thousands of false accusations and convictions of parents. In many cases (Orkneys, Nottingham) children that social workers had convinced themselves were being abused were taken into care against their will - and are now suing those same social services that thought they were looking after them, for child cruelty.
The case of the cruelty committed by catholic priests is a special one. Like children in care all over the world, Irish children were especially vulnerable because they had no social status. Most of the cruelty was simple cruelty, because that moronic organisation known as the Catholic church was incapable of seeing the difference between the unfortunate and the wicked. There were instances of sexual molestation, whether because the priests were already peadophiles who opportunistically sought out that occupation, or who, in their weakness, opportunistically abused those who were without defences, we cannot tell.
The condition of the crime is that children without parents were without status, and so easily preyed upon. But still, this is very much the exception, not the rule. Most men love and care for their own children in the proper sense of the words.