In reality, that's something I thought about a lot.
I had not the luck to meet my grandpa, cancer got him before I was even around, but something came through. See, he fought Nazis and Fascists when the rest of the country disagreed with him. When the neighbours sold him to the Fascist police and he got out when his family corrupted the officers to let him go? He went back fighting. When the Nazis beat down my great-grandmother to get to him? He left the town and went back fighting.. After getting out a prisoner's camp? Still went back fighting. He was also enough of a wise man to explain my father why he did what he did and let him decide if that was the right thing to do on his own. I do like to think I get the point as well.
Violence is not a good thing. I'd never do it if given the choice. But, while I Free Speech is a big deal for me, I can't avoid thinking that when a rethoric is undeniably based on things like hate, hurting others or denying other people their right to live a peaceful life because of whatever reason motivates such rethoric, then the ones that follow it should be stopped before the damage is done.
If you ask me, if there's the chance I can avoid bad stuff to happen and avoid sufference for those that never asked for it only by punching a Nazi (or a variant) today, I'd say it's a fair price. I'd compromise my own morality anytime if that avoids other people getting hurt. There's a point after what you preach becomes so toxic it has no right to be.
"Freedom of Speech" is something that comes only after an even more fundamental right: the "Freedom to Be". If you don't grant the latter to everyone, you lose the benefits of the former