Comments are widely used (and misused IMO) for different reasons. At times they are real answers or elaborations of answers, at others they are used just to express personal impressions on questions or answers. Most useful and interesting comments usually receive a good numbers of up-votes and become part of the answers they relate to. But what about unconstructive or unuseful comments? Why don't we have the possibility of downvoting them instead of having to argue about their unhelpful content. In short, if comments ( like questions and answers) have the dignity of being upvoted or flagged, why not that of being downvoted too.
It's not possible to do this because Stack Exchange administration has decided it's not needed. See this Meta SE post: Allow downvoting comments. So discussion here is unlikely to make a difference.
That said, I agree with Josh61 that it would be beneficial to be able to downvote comments. Flagging only seems appropriate to me for comments that are obviously rude or non-constructive. I don't think flagging is right for comments that present information that at first appears to be correct, but then turns out to be wrong: these require too much subjective evaluation to be flagged for deletion. (Just like incorrect but valid answers.) It also doesn't seem right to delete all comments that present controversial but valid opinions. (And I don't think the mods would do this).
Yes, people can respond to comments like these with new comments, but in order for this to be helpful it has to either convince the original poster to delete the unhelpful comment (which is rare) or gain enough attention and upvotes to appear alongside the unhelpful comment in the collapsed view of comments. The second scenario can be difficult when addressing comments on old or rarely viewed questions.
The following answer only applies to the main site. Meta is expected to have discussion, so comments tend to be treated differently here.
Comments are not like answers or questions. They are not meant to last. If you see a comment that is unhelpful then please flag it so that it may be deleted.
Upvoting of comments is there, ideally, to raise the signal to noise ratio of comments. Comments with up votes will be seen above those without if there are 5 or more comments. Once a comment has been actioned it should be deleted. If you see a comment that has been actioned, then please flag it as obsolete.
If people stopped treating comments as a place to have a chat, then there would be less of a problem. (I mean, we have chat rooms for chat.)
Comments should only be made to comment on how to improve a post. If you feel you are getting into an argument then take it to chat. If you feel like you need to make a joke, take it to chat. Don't be surprised if a comment of yours gets deleted.
Comments don't get downvotes because they shouldn't stick around long enough to need them.