It's not possible to do this because Stack Exchange administration has decided it's not needed. See this Meta SE post: http://meta.stackexchange.com/questions/3615/allow-downvoting-commentsAllow downvoting comments. So discussion here is unlikely to make a difference.
That said, I agree with Josh61 that it would be beneficial to be able to downvote comments. Flagging only seems appropriate to me for comments that are obviously rude or non-constructive. I don't think flagging is right for comments that present information that at first appears to be correct, but then turns out to be wrong: these require too much subjective evaluation to be flagged for deletion. (Just like incorrect but valid answers.) It also doesn't seem right to delete all comments that present controversial but valid opinions. (And I don't think the mods would do this).
Yes, people can respond to comments like these with new comments, but in order for this to be helpful it has to either convince the original poster to delete the unhelpful comment (which is rare) or gain enough attention and upvotes to appear alongside the unhelpful comment in the collapsed view of comments. The second scenario can be difficult when addressing comments on old or rarely viewed questions.