Inciting Violence or calling for violent actions to be taken against those people, or taking physical actions against a group of people, is where I set the isms.
There's dumb and wrong opinions, which everyone has, but if it only affects one person and is never acted upon, I'd argue it's a benign belief, one that doesn't cause harm to others and affects no one else, and so isn't worth the effort of getting offended about. It's more productive to laugh or mock that belief then to actually work to keep it down.
To me, when you start crossing the line is when you start calling for actual physical actions to be taken against people based on their skin color, sex, gender, nationality, or religion. And when a person does these things in the physical world, to harm others, because of what they believe about those other people having less worth and therefore deserving what happened to them.
Words are mostly meaningless, societies which have attempted to wrangle in and control the language in ways to reflect morality have often been on the wrong side of history, and often were more harmful to the cause of defeating discrimination or various isms then actually solving them.
It's actions that make isms so bad, and why it's important to reduce them to a mitigated level. I feel like in this day and age, people are way more sensitive to words then actual actions, which I find baffling. People who murder others over religious views are viewed in circles as more morally acceptable then those who hurt others feelings with a strongly worded paragraph or sentence. It's a time I really have no liking for, and I question why anyone would consider feelings more important then actions in determining what is or isn't an ism.