It is even more complex than that. Countries that engaged in primitive plunder of the colonies, such as Spain or Portugal, did not fare very well in the long run. They were overtaken by England that might have plundered their colonies, but also developed a set of domestic institutions that allowed a reproduction of wealth instead of mere consumption.
Germany, that benefited very little from its colonies (e.g. its brief colonial venture in Tanganyika was a complete fiasco), nonetheless did well vis a vis England that did exploit her colonies - and if it were not for the United States and x-USSR, England would be a German protectorate today.
Russia engaged in a primitive resource extraction from its quasi colonies in Asia, but in 1905 she was defeated by Japan that had no colonies. However, as soon as Russia started building institutions of wealth reproduction, which happened shortly after the 1917 revolution, it quickly became the world superpower.
In the US, the South that engaged in primitive exploitation of slave labor was defeated by the North precisely because the North had more efficient ways of reproducing wealth. The North won the Civil War not because of its tactical superiority (the South was far superior tactically), but because it could out-produce the South in war materiel.
This suggests that primitive plunder of colonies may create short term advantages to those who directly engage in it, but in the long run it is neither a sufficient nor a necessary condition for outperforming other nations.
Wojtek