Good question, which I believe brings us the heart of the debate, which is "What are property rights?"
If we assume that individual property rights precede any other claim, then indeed anything should be returned to the "rightful owner" if it was acquired by means other than by free consent of that owner. However, if we assume as Justin - and I - do that society has prior claims to any property, including individual labor, then of course those claims trump any reparation/return claim.
I do not think that there is any legal system unconditionally accepts any of these two assumptions in their pure forms. Under virtually all systems there is a provision for the priority of public interest over private property claims (eminent domain). The only thing that varies is the specifics, such as the circumstances under which the eminent domain (or its equivalent) is used or compensation for the property claimed. Even nationalization of industry in Eastern Europe after WWII was not mere confiscation (except for property belonging to Germans and Nazi collaborators) but the owners received some (albeit minimal) compensation.
With that in mind, it is quite obvious that all property claims are conditioned on the sanction by the currently binding social/legal system, albeit different systems sanction different property rights. The question of reparations/return is meaningful only under a particular legal/political regime. For example, if the system calls for overtime compensation and I am cheated out of that compensation by my employer, I am entitled to "reparations" under that system. If the system changes, my claim remains valid only if the new system recognizes claims arising under the previous system.
On the other hand, if the "old" system did not sanction a particular claim but a new system does, that does not create an ex post facto "right" without making mockery of the very concept of jurisprudence. I understand that various fascist regimes, including the US, try to legislate things back cf. the recent Oregon amendment granting compensation to land owners for their past loses under zoning regulations. However, this undercuts of the notion of the rule of law - because it makes every law subject to approval or revocation at some later time.
With that in mind, I think the verdict on the notion for reparations for slavery is pretty clear - the notion is based on an absurd attempt to rewrite laws ex post facto - it is mockery of justice. That is a regressive position vis a vis that creating measures to eradicate inequalities under the current legal system (such as the affirmative action). The essence of the affirmative action laws is creating new rights (albeit the rationale for these new rights might be grounded in history) rather than ex-post facto recreating these rights back it times when they did not exist.
Wojtek