If someone is targeted or accused wrongly, that's unfair. But, downvotes help to defend the network and hold back a tidel wave of spam, plagarism and god knows what else.

If people follow guidelines, then they won't be downvoted, simple.