IE6 or not IE6

I'm relatively new to CSS based websites, and I want them to be semantically correct and validate. I find myself spending a lot of time figuring out how to get my websites to render correctly in ie6. I've read many articles about the pros and the cons. What makes the most sense to me is the audience for a particular website. What percentage of them using ie6 would I be turning away? If it is small enough, then I should not worry about them, and just design a clean and effective website. I can't help not worry about that one person who views my website in IE6, and says it "sucks".