Well, Hegel's Master-Slave dialectic makes a good argument why even the slave-owners' point of view also has its disappointments. The master has subordinated the other to his will, thereby showing that he values honour more than mere animal existence (the surrendering slave demonstrating the opposite, that he prefers existence to honour). But the master is only honoured by .... a slave. He has esteeem in the eyes of people for whom he has no esteem, a situation that is ultimately unsatisfactory.
Worse still, the further development of the relation between the two revolves around the slave's meeting of the master's appetites, so that, perversely, the master reduces himself to the animal status of collection of appetites, while the active principle (always in denial of its own agency) is in truth the slave, whose labour transforms the conditions of existence to meet the master's appetites.
Hegel goes on to argue that the actual development of the forms of self-reflection stoicism, cynicism, skepticism and christianity [I might have that list wrong] are all succesively higher attempts to deal with the problem of un-freedom, which seek freedom on the intellectual plain, through doubt about the world, self-abandonment and finally freedom in the other realm of God's Kingdom and so on. Through the slave's reflection on his or her condition of slavery, true freedom, freedom that is more than mere subjugation of the other can be imagined (and after being imagined, realised).
Maybe it is just a pretty tale, but I tend to think there is something to it, however idealised.
Certainly the fundamentally dissatisfying experience of lordship over others who are not themselves worthy enough to enjoy their esteem is a common theme in literature.