So far, the only times we have turned somebody's votes off for flagging have been when they flagged our own accounts.

We consider unfair flagging to be flagging that is politically motivated or vengeful to an extreme.

Flagging somebody because they state a political opinion that you disagree with would be politically motivated (flagging as economic censorship) and we consider that unfair flagging.

If you're downvoted and downvote a similar amount in response, okay. If you're downvoted and try to destroy their account in response, that's extreme.

I don't consider flagging a personal attack (any post with ad hominem attacks - discourse that targets the person, not the facts) to be vengeful or politically motivated, and you're largely in the clear. Although if you try to destroy their account because of it, that's vengeful.