I am a high rated member of money.se and continue to struggle with the downvotes, sometimes for my own answers, more often for others' frustration. SE doesn't require a comment to downvote, and as John said, there's a 'troll' factor here as there is anyplace on the net.
Whether it be for a question or an answer, the Downvote can feel like a personal attack. There's no 'real world' analog to it. I've never seen a person say "you are wrong" and walk away without another word.
Last, the effort to change the code here is probably more trouble than it's worth, and a forced comment-for-downvote would potentially have the unintended consequence of starting debates that should just be avoided.
Last - I'd bet you experience this far less on the board you frequent most, Graphic design, than here, as it takes time to learn the personality each board has, I say, as I brace myself for the DV to this answer.