Cats have lots of funny behaviors. Chief among them is the habit of sitting on their humans' clothes. What is it about our clothes that's irresistible to cats?
Veterinarian Dr. Karen Primm believes this behavior goes back to an instinct to survive. She points out that when wild felines live in a "family group," they share resources and communicate the value of these resources to other members by scent marking.
Because you have (unintentionally) scent marked your clothes, your cat may perceive these clothes as safe or comforting. As Dr. Primm puts it, "your clothes bear your scent and it is like a sign posted for your cat that says, 'I have checked these items out and they are alright.'"
Cats also like to leave their own scent on things they want to mark as "theirs," and lying on your clothes accomplishes this task, in addition to being comforting.
On a more obvious level, your clothes are also probably soft and fairly comfortable. Additionally, positioning themselves on your clothes probably places your cat pretty near to you, which is likely to get them attention. Cats also tend to love anything that resembles a small space or confined area, and something like a sweatshirt might fit that description nicely (or at least, well enough for cats' taste).
So next time you catch your cat getting their white hair all over your black dress, don't be too mad: they're only doing it because they love you (more or less).