> I thought that if things went wrong, and far
> enough, the holder of a swap could in worse case be responsible
> for the notional amount.
This can happen under rare circumstances; it happened in the UK during the push to protect the pound, and once or twice in Sweden: you have to see an inversion of the curve that is so steep as to cause your counterparty to give you a margin call.
Typically these agressive moves (500% overnight rates, etc.) are for a short enough time that no one calls them in.
But in general, swaps aren't a security, they are just an exchange of cash flows. And most of the outstanding notional is in no danger of default.
/jordan