Actually we're not all of the same mind on the issue (of course anyone who takes evolutionary psychology seriously would grant that there was no _single_ environment of evolutionary adaptation--the EEA is a simplification, but also a far more useful simplification than you seem to think). Obviously, though, there are some periods of our shared evolutionary history that our more useful for generating ev psych hypotheses than others--if the goal is to understand e.g. why human males behave differently from human females, better to think about the problems cavemen faced than those of the first fish to have crawled out of the sea.
-- Luke