Somebody: No, it's not a trifle, it's just not remotely comparable to the fear of physical or sexual assault and violation.
Men fear being psychologically devalued by women and fear loss of opportunities for sex. I know that may be laughable to some, but it can be mentally painful. The latter issue is certainly something men have to deal with more than women, all else being equal. How many millions of men out there can never be in a sexual or romantic relationship because they don't know how to attract women due to appearance, economic or family situation, mental state, age, or what have you? Women's problem with men is of a different character. Much more serious, but at the same time, not equivalent.
In addition, when men fear women, they're not simply fearing her but all the men who they know she prefers to them and who could even pose a physical threat. So in a sense, fearing women is part and parcel of fearing other men. Then again perhaps these are trifles, but they're the age-old trifles of mankind's evolutionary history.