At present, users wanting to be courteous leave comments after down-voting a response, but there is nothing that requires a user to do this.
I don't much care about the lose of rep. or the down-vote itself, but it doesn't give me much insight into how a question/response can be improved. Why is there nothing designed to require people to leave comments on down-votes to make their thoughts known and hold them to a level of courtesy?
In short, it is by design that downvotes and comments are not associated. The voting function is supposed to be anonymous; also, often there isn't something insightful for every downvoter to add in comments.
I don't know if any body has noticed, but I think it is rare to non-existent for anyone to give an explanation for a downvote. My explanation:
you don't have to explain your upvotes, why do you have to go to the bother of doing it for downvotes?
downvotes are already pretty rare anyway (look at anybody's account). This means that they are really stigmatized by the voter for whatever reason (I take it as a given that it feels very hurtful by the recipient). An explanation is not anonymous and so much more easily retaliated, however reasonable the downvote and explanation might be.
So there is a large psychological cost to the explanation, only some to the actual vote. Forcing an explanation would result in few down votes altogether.
There is far too much downvoting without valid reasons. I'd advocate that:
Anyone downvoting should have to provide an explanation. That explanation should be published as an anonymous comment by the system, or provided anonymously to the person whose answer has been downvoted.
Ideally, high-reputation users should be able to review an anonymous list of downvotes (for other people's answers only) and associated reasons, and veto the downvote if no remotely valid reason has been specified.
Downvoting an answer should reduce your reputation by 20 points instead of just one or two, or perhaps by 1% of your points total instead of by a fixed number of points.