After the Neo-Nazi and white supremacist rally in Charlottesville, Virginia, last weekend, a lot of my friends took to Facebook, Twitter, and Tumblr to express their outrage. Outrage is gratifying to express, if not all that helpful – my article last week discussed ways in which the response to Charlottesville has been lacking, and why that’s frightening and alienating to some of the people most affected by it. Setting that aside, the main responses to the events have been guides on how to successfully punch your local Nazi and why we should make an exception to the First Amendment when it comes to hate speech.

Our free speech laws are pretty clear.

According to the American Bar Association, while the First Amendment doesn’t prohibit speech that a listener might find hurtful, wrong, or offensive, it does prohibit “fighting words” — that is, statements that could reasonably provoke an individual to react violently, such as racial or ethnic slurs.

“Fighting words” are one thing. Hate speech is another.

In the past, the Supreme Court has struck down state laws prohibiting hate speech, as legislating the intention behind someone’s words is prohibited under the First Amendment. In the United States, only the action is punishable, not the specific intent behind it.

Most of the people I know find this utterly reprehensible. They believe that by expressing views they see as racist, offensive, and violent, the marchers in Charlottesville and white supremacists across the country are forfeiting their right to free speech. They would like to see hate speech outlawed and punishable by prison time, or better yet, they’d like hate speech to be a community matter, one that can be dealt with by finding a local Trump supporter and beating them with a baseball bat – and facing no legal repercussions for their actions.

That’s not what I want.

That may come as a surprise to some people, since I’m Jewish, and Jewish people were among the principal targets of the Charlottesville marchers. I’d love it if there were a way to silence Nazis without undermining the First Amendment.

But there isn’t. Because whatever limits we place on free speech can be used against us.

Right now, the cultural conversation is shifting leftwards, toward tolerance and away from the right wing. But history is a pendulum, and should the pendulum swing back toward the right, the same limits we’ve placed on free speech can be levied against us. One of the first actions a fascist society takes is limiting free speech. When Hitler came to power in Germany, Joseph Goebbels, his Minister of Propaganda, informed the press that in order to continue to write, they had to be members of his press organization, and that any articles critical of the Third Reich would be considered acts of treason. Not only did the Third Reich enforce these rules on the press, they enforced them on individual citizens, by encouraging individuals to inform on their neighbors and often to take matters into their own hands.

I’m Jewish, and I’m against the types of hate speech legislation that most leftists I know are eagerly espousing. It may seem attractive in the short term, but in the long term, it’s more damaging to our society than a few hundred Neo-Nazis marching in Virginia. In my view of things, ant-Semitism and racial hatred have been around for thousands of years.

Let them talk.

I’d rather know exactly who I’m dealing with.