I think it's got something to do with the idea that most people have that the tax system is progressive: the more you make, the higher your taxes will be. If only that was true! Someone making $40k/yr doesn't see that if they just made $100k, their taxes would actually be "less" because they'd have so much more incremental net income.
So they say: I hate *my* taxes, I can't imagine how awful taxes are for someone making twice what I do!
(of course you're not rich in the US until you're over about $500k/yr of income, but we digress)
/jordan