This is dubious. I had thought that the reason why child rape was considered perverted was because it is a wicked and cruel thing to torture children in that way, not because 'most men' 'cannot imagine' being attracted to a child. First of all, it is actually impossible to say how many men find children attractive, because beyond those who actually proceed from desire to rape, who else is going to make their desires known? Secondly, even among offenders, how many are actually known about? It is estimated that most such rape takes place within families, where it can remain secret and even protected by the family - the recent 'Italian Fritzl' case may be an instance of this. Thirdly, for your argument to work, you would have to assume that all child abusers meet the criteria for a clinical diagnosis of paedophilia, which would be an inaccurate assumption. It may also be that not every paedophile acts on his (or in the rare case, her) desire.
Finally, the whole basis of your objection is laughably reactionary. Sexuality is socially produced, and there is no dearth of evidence that large numbers of men have been attracted to children throughout history. In fact, there are certain supposedly 'harmless' forms of paedophilia that are culturally quite common, such as the 'Lolita' and 'schoolgirl' child-woman fixations: for example, the multinationals that fashioned the Britney and JoJo personas presumably knew what they were doing. Racks and racks of perfectly legal and saleable pornographic magazines contain fantasies that allude to child-fucking, incest and so on: 'barely legal' and 'sweet tens', 'daddy's little girl' etc. British tabloids love to show you a page 3 stunner who is only just legal: the Daily Star once made itself notorious by featuring a girl who was covering her nipples, with the disclaimer: "she's only fifteen, and when she's sixteen, we'll show you her nipples". The sexualisation of children in culture is hard to miss. While I'm on the subject, gay pornography often stars 'twinks' - young men who look extremely boyish and almost prepubescent, because they have little body hair and so on. I am not pretending that all of this should be equated with actual child pornography, in which what is enjoyed is the real life rape and torture of children. What I am suggesting is that the evidence would appear to be that at least millions of men - men who aren't child rapists - are prepared to entertain such desires. Any attempt to base the censure of child rape on assuptions about the 'normal' workings of male desire in a patriarchical society can only succeed by reproducing discourses of deviancy, discourses that medicalise sexual desire (bear in mind that paedophilia is actually treated as a subcategory of 'paraphilia' which is the usually harmless but apparently 'sick' desire aroused by a particular object), and which inevitably play into homophobic and misogynistic stereotypes.
Perhaps what you meant to say, or should have said, was that most men are repelled by the idea of torturing children - which, even if we can't prove it, is a thesis I am willing to put my hope in.