The point with someone like Polanski is, he did it once and isn't doing it any more, so your support or otherwise doesn't perpetuate injustice (it won't make a difference to whether he returns to the US or not).
Whereas, say, with certain internet porn sites you could be looking at trafficked women and in such a scenario you are part of an ongoing problem (supply/demand, etc). Issues like that are the issues I prefer to get indignant about, if I'm going to get indignant about anything at all. Slightly different issue, I know, but if I worry about supporting anything, it will be injustices of that kind.
But just to be clear: We are all different. I certainly don't necessarily expect everyone to adopt my extremist view on art versus artist.